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ABSTRACT 



A force feedback system provides components for use in a 
force feedback system including a host computer and a force 
feedback interface device. An architecture for a host com- 
puter allows multi-tasking application programs to interface 
with the force feedback device without conflicts, where a 
single active application may output forces. A background 
application also provides force effects to be output and 
allows a user to assign force effects to graphical objects in 
a graphical user interface. Force feedback effects and struc- 
tures are further described, such as events and enclosures. 
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FORCE FEEDBACK SYSTEM INCLUDING 

MULTI-TASKING GRAPHICAL HOST 
ENVIRONMENT AND INTERFACE DEVICE 

This invention was made with Government support 
under Contract Number F41624-96- C-6029, awarded by the 
Department of Defense. The Government has certain rights 
in this invention. 

BACKGROUND OF THE INVENTION 

The present invention relates generally to interface 
devices for allowing humans to interface with computer 
systems, and more particularly to computer interface devices 
that allow the user to provide input to computer systems and 
provide force feedback to the user. 

Computer systems are used extensively to implement 
many applications, such as word processing, data 
management, simulations, games, and other tasks. A com- 
puter system typically displays a visual environment to a 
user on a display screen or other visual output device. Users 
can interact with the displayed environment to perform 
functions on the computer, play a game, experience a 
simulated environment, use a computer aided design (CAD) 
system, etc. One visual environment that is particularly 
common is a graphical user interface (GUI). GUI's present 
visual images which describe various graphical metaphors 
of a program or operating system implemented on the 
computer. Common operating systems using GUI's include 
the Windows™ operating system from Microsoft Corpora- 
tion and the MacOS operating system from Apple Computer, 
Inc. The user typically moves a displayed, user-controlled 
graphical object, such as a cursor or pointer, across a 
computer screen and onto other displayed graphical objects 
or predefined screen regions, and then inputs a command to 
execute a given selection or operation. The objects or 
regions ("targets") can include, for example, icons, 
windows, pull-down menus, buttons, and scroll bars. Most 
GUI's are currently 2-dimensional as displayed on a com- 
puter screen; however, three dimensional (3-D) GUI's that 
present simulated 3-D environments on a 2-D screen can 
also be provided. Other programs or environments that may 
provide user-controlled graphical objects such as a cursor or 
a "view" controlled by the user include graphical "web 
pages" or other environments offered on the World Wide 
Web of the Internet, CAD programs, video games, virtual 
reality simulations, etc. 

The user interaction with and manipulation of the com- 
puter environment is achieved using any of a variety of types 
of human-computer interface devices that are connected to 
the computer system controlling the displayed environment. 
In most systems, the computer updates the environment in 
response to the user's manipulation of a user-manipulatable 
physical object ("user object") that is included in the inter- 
face device, such as a mouse, joystick, etc. The computer 
provides feedback to the user utilizing the display screen. A 
computer mouse is a common user object used to interact 
with a GUI or other graphical environment. A mouse (and 
other mouse-type devices such as a track ball) is typically 
used as a position control device in which displacement of 
the mouse in a planar workspace (e.g. on a mouse pad) is 
directly correlated to displacement of the user-controlled 
graphical object, such as a cursor, displayed on the screen. 

Force feedback interface devices allow a user to experi- 
ence forces on the manipulated user object based on inter- 
actions and events within the displayed graphical environ- 
ment. Typically, computer-controlled actuators are used to 
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output forces on the user object in provided degrees of 
freedom to simulate various sensations, such as an obstruc- 
tion force when moving the cursor into a wall, a damping 
force to resist motion of the cursor, and a spring force to bias 

5 the cursor to move back toward a starting position of the 
spring. Force feedback devices can be implemented in many 
forms, such as a joystick, mouse, steering wheel, etc. 

When implementing force feedback sensations in a GUI 
of an operating system, several problems can arise. One 

10 problem is the use of force feedback when multiple appli- 
cation programs are simultaneously running in a multi- 
tasking environment on the host computer. Most operating 
systems allow such multi-tasking, for example, to allow a 
user to interact with one application while one or more 

15 applications are also running, receiving data, outputting 
data, or performing other tasks. For example, in the Win- 
dows™ operating system, one application is the "active" 
application that typically displays an active window in the 
GUI. The user can manipulate the functions of the active 

20 application using the cursor. Other inactive applications are 
also running and may have inactive windows displayed in 
the GUI. The user can switch to a different application by 
clicking the cursor in an inactive window, for example, 
which causes the new application to be the active application 

25 and the formerly active application to become inactive. 
Each application run by an operating system may have its 
own set of force sensations that it needs to command to the 
force feedback device. Thus, one application may need to 
command spring, force, vibration, and texture force 

30 sensations, while a different application may need to com- 
mand spring, damper, and jolt force sensations. The force 
feedback device typically cannot store all possible force 
sensations for each application running in the operating 
system, so there is a problem of which force sensations the 

35 force feedback device should store and implement at any 
one time. In addition, if two of the multi-tasking applications 
command conflicting force sensations, the force feedback 
device needs to choose one of the force sensations to output, 
and there currently is no system or method of doing so. 

40 A different problem occurs when using a force feedback 
device with a GUI. Traditional mouse controllers used with 
GUI's are relative position reporting devices, i.e., they 
report only changes in position of the mouse to the host 
computer, which the host computer uses to calculate a new 

45 position for the cursor on the screen. Many force feedback 
devices, in contrast, are typically absolute position reporting 
devices which report an absolute position of the cursor, such 
as screen coordinates, to the host computer. This is because 
the force feedback device needs to know the cursor position 

50 to accurately determine when forces are to be applied and to 
accurately calculate the forces. However, it would be desir- 
able in some instances to have a relative position reporting 
force feedback device, since the host computer is standard- 
ized to receive and interpret relative positions at the most 

55 basic level. Furthermore, such a relative device would 
permit the host computer to perform needed adjustments to 
cursor position, such as ballistics calculations which modify 
cursor position based on mouse velocity to provide 
enhanced control. If the host computer performs such 

60 adjustments, the force feedback device processors are 
relieved of computational burden. In addition, some types of 
interface devices such as trackballs are better suited to 
relative position reporting since an absolute, limited work- 
space is not easily defined for these devices. 

65 Another problem occurs when force feedback is imple- 
mented with a GUI or other graphical environment and the 
graphical environment changes resolution or aspect ratio. 
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For example, if a resolution of 640x480 is being displayed enabled, and provides absolute positions to the host when 

by the host computer on a screen, the force feedback device force feedback is enabled. Sensor data is read and a position 

assumes that graphical objects in the GUI have a size of a user manipulandum in a device frame is determined. A 

proportional to screen dimensions and outputs forces device delta position of the manipulandum includes the 

accordingly. However, when the resolution is changed, the 5 change in position of the manipulandum from a previous 

objects displayed on the screen change size in proportion to position and is reported to the host computer when the host 

the screen dimensions. The force feedback device continues computer is not enabled to receive an absolute screen 

to check for conditions and generate forces as if the old P 05 ^ 00 of me CUIS0 / from the force ^dback device. When 

resolution were active, resulting in forces that do not cor- host 15 eaa ^ led t0 receive th f cursor 

relate with displayed interactions on the screen. The aspect 10 P° sltIon > . the Posmonis determined from the device 

ratio of a display screen can also change, e.g., when two ddta Position and is reported to the host computer to allow 

screens are used to provide double the amount of displayed said L host ^mputer to display the cursor in the screen frame 

area in the GUI, the aspect ratio doubles in one dimension. at th „ e ^ ea Potion A scaled position or "ballistic posi- 

Using prior art force feedback devices, forces can become of the manipulandum related to the screen position of 

distorted from such an aspect ratio change. For example, a is cursor is used in determining a force to be output by 

circle object displayed on the screen may have forces at its actuators of the force feedback device. Preferably, the scaled 

borders that feel like an ellipse to the user of the interface P oslt ! on * determined from the delta position and the screen 

device, since the aspect ratios of the screen and the mouse P osU ! on °1 sf^.^sor is determined based on toe scaled 

workspace are different position. Ballistic parameters and screen size of the host 

20 display screen can be received from the host computer to 

SUMMARY OF THE INVENTION determine the scaled position and screen position. 

In another aspect of the present invention, a force feed- 

The present invention is directed to a force feedback back device, such as a force feedback mouse, provides only 

system and architecture which allow control of a force relative positions to a host computer. A local microprocessor 

feedback device in a multi-tasking graphical host environ- 25 on tne force feedback device determines a delta position of 

ment. In addition, force feedback device embodiments are ^ t manipulandum that includes the change in position of the 

disclosed which provide for relative position reporting and manipulandum from a previous position, and the delta 

absolute position reporting to the host computer. position is reported to the host. A screen position of the 

More specifically, a method of the present invention cursor is modeled from the delta position and the modeled 
interfaces a multi- tasking graphical environment imple- 30 screen position is used in the determination of a force output 
mented on a host computer with a force feedback interface by actuators of the device. An actual screen position of the 
device coupled to the host computer, where multiple appli- cursor is received from the host computer and the modeled 
cation programs may run in the multi-tasking environment. screen position is corrected based on the actual screen 
A context in created for association with each application position. Preferably, the modeled screen position is cor- 
program running in the multi-tasking graphical environ- 3S rected incrementally over time to smooth out the correction 
ment. Force effect commands are received from the appli- and reduce a perception of the error correction by the user, 
cation programs, where the force effect commands com- Preferably, both the host and the device adjust the screen 
mand the force feedback interface device to output a force position using a ballistics algorithm. The host also prefer- 
effect specified by the command. The force effect commands ably sends ballistic parameters and screen size to the device 
are stored into the contexts. Each context is associated with ^ to allow accurate modeling of the screen position, 
one of the application programs running on the host [ Q another aspect of the present invention, a force feed- 
computer, and each force effect command is stored in a back device reads sensor data and determines a position of 
context associated with the application program that sent the a user manipulandum in a workspace of the device. A 
force effect command. The force effect commands in the position of the manipulandum is reported to the host com- 
context of a particular application program are sent to the 45 pu ter so that the host can display the cursor in a graphical 
force feedback device when that particular application pro- environment. Information is received from the host corn- 
gram is active in the multi-tasking environment. Preferably, puter describing a current screen size of the display screen 
the force effect commands in contexts of inactive application 0 f the host computer, where the current screen size is 
programs are not sent to the force feedback device. Thus, included in a determination of the position reported to the 
only the active application program may command forces on 50 host. 

the force feedback device. Also described herein are several force feedback sensa- 

When the application program becomes inactive and a tions and structures, including enclosures, textures, and 

new application program becomes active, new force effect grids. For example, an enclosure is a rectangular or elliptical 

commands are sent to the force feedback device to replace shape having walls that provide forces to an associated 

the force effect commands of the formerly active applica- 55 graphical object such as an icon or window. The enclosure 

tion. Preferably, a background application is provided which includes walls, where each wall may be associated with a 

also provides force effects to the force feedback device and force. In one embodiment, a particular type of enclosure is 

may output forces on the device even when not active. provided to modify the forces of a different enclosure to 

Events are also provided which allow a graphical action prevent conflicts between the forces of overlapping or 

such as an interaction of the cursor in the graphical envi- 60 closely-positioned enclosures. 

ronment with another object to cause an event notification to The present invention provides several embodiments of 
be sent to the application program. Preferably, only the components in a force feedback system. The architecture on 
active and background application programs may receive the host computer allows multi-tasking application pro- 
events, grams to interface with the force feedback device without 
In another aspect of the present invention, a force feed- 65 conflicts. The force feedback device provides both relative 
back device, such as a force feedback mouse, provides position reporting and absolute position reporting to allow 
relative positions to a host when force feedback is not great flexibility. A relative position reporting device allows 
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maximum compatibility with existing software. Information user. By "grasp, M it is meant that users may releasably 

such as ballistic parameters and screen size sent from the engage a portion of the object in some fashion, such as by 

host to the force feedback device allow accurate mouse hand, with their fingertips, etc. For example, images are 

positions and cursor positions to be determined in the force displayed and/or modified on a display screen 20 of the 

feedback environment. 5 computer system 18 in response to such manipulations. In 

™, j *u j * r *l * - *■ -ii the described embodiment, user object 12 is a mouse 12 that 

These and other advantages of the present invention will • « j 4t _ * , * uj c _* t_i 

• 4 , , ... , . (L r ^ j. - is shaped so that a user s fingers or band may comfortably 

become apparent to those skilled in the art upon a reading of _ , - , „, • • / . , ^ J f 

. i- ii • *> t_ • . . j * grasp the object and move it in the provided degrees of 

the following specification of the invention and a study of freedom ^ physical space Fof a ^ can mQve 

the several figures of the drawing. ^ mouse n tQ amsiQutia$y move a computer generated 

BRIEF DESCRIPTION OF THE DRAWINGS graphical object, such as a cursor or other image, in a 

graphical environment provided by computer 18. The avail- 

FIG. 1 is a perspective view of one embodiment of a able degrees of freedom in which mouse 12 can be moved 

mouse interface system suitable or use with the present a re determined from the interface 14, described below. In 

invention; ^ addition, mouse 12 preferably includes one or more buttons 

FIG. 2 is a perspective view of an embodiment of a 15 to allow the user to provide additional commands to the 

mechanism suitable for the interface system of FIG. 1; computer system. 

FIG. 3 is a block diagram of the system of FIG. 1 for It will be appreciated that a great number of other types 

controlling a force feedback interface device of the present of user manipulable objects can be used with the method and 

invention; 20 a PP aratus of the present invention in place of or in addition 

FIG. Vis a block diagram of an architecture for a host to mouse 12 For example, such objects may include a 

computer providing multiple application programs commu- s P here > * P uck > a joystick, cubical- or other-shaped hand 

nicating with the force feedback device; g"**' a receptacle for receiving a finger or a stylus, 

„ „ _ . . .„ x . r t_ i j a flat planar surface like a plastic card having a rubberized, 

FIG. 5 is a diagrammatic lUustration of a background COQto i red , and/orbum p y J rface , ahandh eW remote control 

application program control panel allowing a user to char- * ^ for controlling web pagps or other devices> or other 

actenze background torces; objects. In one embodiment, a small fingertip joystick can be 

FIGS. 6a, 6b and 6c are diagrammatic illustrations of provided, where a small stick is moved in a small planar 

embodiments of an enclosure force effect; region with a user's fingertips. Spring forces can be provided 

FIG. 7 is a diagrammatic illustration of a texture force 3Q by the actuators of the device 11 to bias the stick (or any type 

effect; of joystick) toward the center of the planar region to 

FIG. 8 is a diagrammatic illustration of a grid force effect; simulate a spring return on the joystick. Since fingertips are 

FIG. 9 is a block diagram illustrating a preferred embodi- used > 0Ut P ut forces need not 06 ^ hi &* a magnitude as in 

ment of implementing the force feedback device and system other embodiments. Also, mouse 12 can be provided with 

of the present invention; 35 such a centering spring bias, e.g. when used like a joystick 

FIG. 10 is a flow diagram illustrating a method of *° S amm 8 applications, 

implementing position reporting for the embodiment of FIG. Interface 14 mterfaces mechanical and electrical input and 

g. output between the mouse 12 and host computer 18 unple- 

'„ n - „ j. . ,. „. , , , A mentine the application program, such as a GUI, simulation 

FIG. 11 is a now diagram illustrating a method of deter- ° . /, , » tA - , t4 . , 

& , . , . * , ... A - xn or game environment. Interface 14 provides multiple 

mimng cursor position and indexing in the embodiment of w , 6 rrj * ii • *u * j 

FIGS 9 and 10* degrees or freedom to mouse 12; in the preferred 

' embodiment, two linear, planar degrees of freedom are 

FIG. 12 is a block diagram illustrating a second embodi- pTOviM to the mouse , „ shown by arrows 2 2. In other 

ment of implementing the force feedback device and system embodiments, greater or fewer degrees of freedom can be 

of the present invention; 45 provided, as well as rotary degrees of freedom. For many 

FIG. 13 is a flow diagram illustrating a method of applications, mouse 12 need only be moved in a very small 

implementing position reporting for the embodiment of FIG. workspace area. 

12> anc * In a preferred embodiment, the user manipulates mouse 

FIGS. 14 and 15 are diagrammatic illustrations showing 12 in a planar workspace, much like a traditional mouse, and 

the error correction between cursor positions from host 50 the position of mouse 12 is translated into a form suitable for 

computer and force feedback device in the embodiment of interpretation by position sensors of the interface 14. The 

FIG. 13. sensors track the movement of the mouse 12 in planar space 

DETAILED DESCRIPTION OF PREFERRED and provide smtable el^ronic signak to an electronic 

EMBODIMENTS portion of interface 14. The interface 14 provides position 

55 information to host computer 18. In addition, host computer 

FIG. 1 is a perspective view of a force feedback mouse 18 and/or interface 14 provide force feedback signals to 

interface system 10 of the present invention, capable of actuators coupled to interface 14, and the actuators generate 

providing input from the user to a host computer based on forces on members of the mechanical portion of the interface 

the user's manipulation of the mouse and capable of pro- 14 to provide forces on mouse 12 in provided or desired 

viding force feedback to the user of the mouse system based 60 degrees of freedom. The user experiences the forces gener- 

on events occurring in an environment implemented by the ated on the mouse 12 as realistic simulations of force 

host computer. Mouse system 10 includes an interface sensations such as jolts, springs, textures, enclosures, 

device 11 that includes a user manipulatable object or circles, ellipses, grids, vibrations, "barrier" forces, and the 

manipulandum 12 and an interface 14, and a host computer like. 

18. 65 The electronic portion 26 of interface 14 may couple the 

User object (or "manipulandum") 12 is an physical object mechanical portion 24 of the interface to the host computer 

that is preferably grasped or gripped and manipulated by a 18. The electronic portion 26 is preferably included within 
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the housing 21 of the interface 14 or, alternatively, the 
electronic portion may be included in host computer 18 or as 
a separate unit with its own housing. More particularly, 
interface 14 includes a local microprocessor distinct and 
separate from any microprocessors in the host computer 18 
to control force feedback on mouse 12 independently of the 
host computer, as well as sensor and actuator interfaces that 
convert electrical signals to appropriate forms usable by the 
mechanical portion of interface 14 and host computer 18. 

For example, a rigid surface is generated on computer 
screen 20 and a computer object (e.g., cursor) controlled by 
the user collides with the surface. In a preferred 
embodiment, high-level host commands can be used to 
provide the various forces associated with the rigid surface. 
The local control mode using a local microprocessor in 
interface 14 can be helpful in increasing the response time 
for forces applied to the user object, which is essential in 
creating realistic and accurate force feedback. For example, 
it is preferable that host computer 18 send a "spatial repre- 
sentation" to the local microprocessor, which is data describ- 
ing the locations of some or all the graphical objects 
displayed in a GUI or other graphical environment which are 
associated with forces and the types/characteristics of these 
graphical objects. The microprocessor can store such a 
spatial representation in local memory, and thus will be able 
to determine interactions between the user object and 
graphical objects (such as the rigid surface) independently of 
the host computer. In addition, the microprocessor can be 
provided with the necessary instructions or data to check 
sensor readings, determine cursor and target positions, and 
determine output forces independently of host computer 18. 
The host could implement program functions (such as 
displaying images) when appropriate, and synchronization 
commands can be communicated between the microproces- 
sor and host 18 to correlate the microprocessor and host 
processes. Such commands and related functionality is dis- 
cussed in greater detail below. Alternatively, the computer 
18 can directly send force feedback signals to the interface 
14 to generate forces on mouse 12. A suitable embodiment 
of the electrical portion of interface 14 is described in detail 
with reference to FIG. 3. 

The interface 14 can be coupled to the computer 18 by a 
bus 17, which communicates signals between interface 14 
and computer 18 and also, in the preferred embodiment, 
provides power to the interface 14 (e.g. when bus 17 
includes a USB interface). In other embodiments, signals 
can be sent between interface 14 and computer 18 by 
wireless transmission/reception. In preferred embodiments 
of the present invention, the interface 14 serves as an 
input/output (I/O) device for the computer 18. The interface 
14 can also receive inputs from other input devices or 
controls that are associated with mouse system 10 and can 
relay those inputs to computer 18. For example, commands 
sent by the user activating a button on mouse 12 can be 
relayed to computer 18 by interface 14 to implement a 
command or cause the computer 18 to output a command to 
the interface 14. 

Host computer 18 is preferably a personal computer or 
workstation, such as an IBM-PC compatible computer or 
Macintosh personal computer, or a SUN or Silicon Graphics 
workstation. For example, the computer 18 can operate 
under the Windows™ or MS-DOS operating system in 
conformance with an IBM PC AT standard. Alternatively, 
host computer system 18 can be one of a variety of home 
video game systems commonly connected to a television set, 
such as systems available from Nintendo, Sega, or Sony. Id 
other embodiments, host computer system 18 can be a "set 
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top box" which can be used, for example, to provide 
interactive television functions to users, or a "network-" or 
"internet-computer" which allows users to interact with a 
local or global network using standard connections and 

5 protocols such as used for the Internet and World Wide Web. 
Host computer preferably includes a host microprocessor, 
random access memory (RAM), read only memory (ROM), 
input/output (I/O) circuitry, and other components of com- 
puters well-known to those skilled in the art. 

iq Host computer 18 preferably implements one or more 
application programs ("applications") with which a user is 
interacting via mouse 12 and other peripherals, if 
appropriate, and which can include force feedback function- 
ality. For example, the host application programs can 

! 5 include a simulation, video game, Web page or browser that 
implements HTML or VRML instructions, word processor, 
drawing program, spreadsheet, scientific analysis program, 
or other application program that utilizes input of mouse 12 
and outputs force feedback commands to the mouse 12. 

20 Typically, an operating systems such as Windows™, 
MS-DOS, MacOS, Unix, is also running on the host com- 
puter and preferably includes its own force feedback func- 
tionality. In one preferred embodiment, the operating system 
and application programs utilize a graphical user interface 

25 (GUI) to present options to a user, display data and images, 
and receive input from the user. In the preferred 
embodiment, multiple applications can run simultaneously 
in a multitasking environment of the host computer, as is 
detailed below. Herein, computer 18 may be referred as 

30 displaying "graphical objects" or "computer objects." These 
objects are not physical objects, but are logical software unit 
collections of data and/or procedures that may be displayed 
as images by computer 18 on display screen 20, as is well 
known to those skilled in the art. A displayed cursor or a 

35 simulated cockpit of an aircraft might be considered a 
graphical object. The host application program checks for 
input signals received from the electronics and sensors of 
interface 14, and outputs force values and/or commands to 
be converted into forces on mouse 12. Suitable software 

40 drivers which interface such simulation software with com- 
puter input/output (I/O) devices are available from Immer- 
sion Human Interface Corporation of San Jose, Calif. 

Display device 20 can be included in host computer 18 
and can be a standard display screen (LCD, CRT, etc.), 3-D 

45 goggles, or any other visual output device. Typically, the 
host application provides images to be displayed on display 
device 20 and/or other feedback, such as auditory signals. 
For example, display screen 20 can display images from a 
GUI. Images describing a moving, first person point of view 

50 can be displayed, as in a virtual reality game. Or, images 
describing a third-person perspective of objects, 
backgrounds, etc. can be displayed. Alternatively, images 
from a simulation, such as a medical simulation, can be 
displayed, e.g., images of tissue and a representation of a 

55 manipulated user object 12 moving through the tissue, etc. 
There are two primary "control paradigms" of operation 
for mouse system 10: position control and rate control. 
Position control is the more typical control paradigm for 
mouse and similar controllers, and refers to a mapping of 

60 mouse 12 in which displacement of the mouse in physical 
space directly dictates displacement of a graphical object. 
Under a position control mapping, the computer object does 
not move unless the user object is in motion. Also, "ballis- 
tics" or other non-linear adjustments to cursor position can 

65 be used in which, for example, slow motions of the mouse 
have a different scaling factor for cursor movement than fast 
motions of the mouse, to allow more control of short cursor 
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movements. Several different ways of implementing ballis- as a cursor, in the host frame displayed by host computer 18. 

tics and other control adjustments in a force feedback device In one implementation, the user may reposition the mouse 

are described in co-pending patent application Ser. No. 12 without providing any input to the host computer, thus 

08/1924,462, and these adjustments can be used in mouse allowing the user to redefine the offiset between the object's 

system 10 if desired. 5 position and the cursor's position. Such indexing is achieved 

As shown in FIG. 1, the host computer may have its own through an input device such as a button, switches, sensors, 

"screen frame" 28 (or host frame) which is displayed on the or otner devices. As long as the indexing device is 

display screen 20. In contrast, the mouse 12 has its own activated, the mouse 12 is in indexing mode; when the 

"device frame" (or local frame) 30 in which the mouse 12 is button is released (or indexing mode otherwise exited), the 

moved. In a position control paradigm, the position (or 10 position of the cursor is again controlled by the mouse 12. 

change in position) of a user-controlled graphical object, A hand weight switch can also be provided which inherently 

such as a cursor, in host frame 28 corresponds to a position causes indexing when the user removes hand or finger 

(or change in position) of the mouse 12 in the local frame 30. wei 8 ht from mouse 12 In ooe embodiment, the functionality 

Rate control is also used as a control paradigm. This refers of a ^/«?tch and the indexing mode are . integrated into 

to a mapping in which the displacement of the mouse 12 is 15 °« ^^^^^^"tetri m parent US. 

abstractly mapped to motion of a computer object under j£ No ' 5 ' 825 ' 308 and P atcnt Ration Ser. No. 08/881, 
control. There is not a direct physical mapping between 

physical object (mouse) motion and computer object A different way to allow indexing is to provide a com- 
motion. Thus, most rate control paradigms are fundamen- bined position control and rate control device which allows 
tally different from position control in that the user object 20 different forms of control of the cursor depending on the 
can be held steady at a given position but the controlled position of the mouse in its workspace. This embodiment is 
computer object is in motion at a commanded or given described in greater detail below and in co-pending patent 
velocity, while the position control paradigm only allows the application 08/924,462, by Rosenberg et al, filed 8/23/97, 
controlled computer object to be in motion if the user object wnich is hereby incorporated by reference herein, 
is in motion. 25 FIG. 2 is a perspective view of a preferred embodiment of 

The mouse interface system 10 is useful for both position the mouse device 11 with the cover portion of housing 21 

control ("isotonic") tasks and rate control ("isometric") and the grounded pad 32 removed. Mechanical linkage 40 

tasks. For example, as a traditional mouse, the position of provides support for mouse 12 and couples the mouse to a 

mouse 12 in its local frame 30 workspace can be directly grounded surface 34, such as a tabletop or other support, 

mapped to a position of a cursor in host frame 28 on display Linkage 40 is, in the described embodiment, a 5-member (or 

screen 20 in a position control paradigm. Alternatively, the "5-bar") linkage. Fewer or greater numbers of members in 

displacement of mouse 12 in a particular direction against an the linkage can be provided in alternate embodiments, 

opposing output force can command rate control tasks in an Ground member 42 of the linkage 40 is a base for the 

isometric mode. An implementation that provides both iso- 35 support of the linkage and is coupled to or resting on a 

tonic and isometric functionality for a force feedback con- ground surface 34. The ground member 42 in FIG. 2 is 

trailer and which is very suitable for the interface device of shown as a plate or base that extends under mouse 12. The 

the present invention is described in U.S. Pat. No. 5,825,308, members of linkage 40 are rotatably coupled to one another 

incorporated by reference herein. through the use of rotatable pivots or bearing assemblies, all 

Mouse 12 is preferably supported and suspended above a w referred to as "bearings" herein. Base member 44 is rotat- 

grounded pad 32 by the mechanical portion of interface 14, ably coupled to ground member 42 by a grounded bearing 52 

described below. Pad 32 or similar surface is supported by and can rotate about an axis A. Link member 46 is rotatably 

grounded surface 34. Pad 32 (or alternatively grounded coupled to base member 44 by bearing 54 and can rotate 

surface 34) provides additional support for the mouse and about a floating axis B, and base member 48 is rotatably 

relieve stress on the mechanical portion of interface 14. In 45 coupled to ground member 42 by bearing 52 and can rotate 

addition, a wheel, roller, Teflon pad or other device can be about axis A. Link member 50 is rotatably coupled to base 

used to support the mouse. member 48 by bearing 56 and can rotate about floating axis 

Mouse 12 can be used, for example, to control a C, and link member 50 is also rotatably coupled to link 

computer-generated graphical object such as a cursor dis- member 46 by bearing 58 such that link member 50 and link 

played in a graphical computer environment, such as a GUI. 5 o member 46 ma y rotate relative t0 each other about floalin g 

The user can move the mouse in 2D planar workspace to ax * s 

move the cursor to graphical objects in the GUI or perform Linkage 40 is formed as a five-member closed-loop chain, 

other tasks. In other graphical environments, such as a Each member in the chain is rotatably coupled to two other 

virtual reality video game, a user can be controlling a members of the chain to provide mouse 12 with two degrees 

computer player or vehicle in the virtual environment by 55 of freedom, i.e., mouse 12 can be moved within a planar x-y 

manipulating the mouse 12. The computer system tracks the workspace. Mouse 12 is coupled to link members 46 and 50 

position of the mouse with sensors as the user moves it. The by rotary bearing 58. and may rotate at least partially about 

computer system may also provide force feedback com- axis D. A pad or other support can be provided under mouse 

mands to the mouse, for example, when the user moves the 12 to help support the mouse. 

graphical object against a generated surface such as an edge 50 Transducer system 41 is used to sense the position of 

of a window, a virtual wall, etc. It thus appears and feels to mouse 12 in its workspace and to generate forces on the 

the user that the mouse and the graphical object are con- mouse 12. Transducer system 41 preferably includes sensors 

tacting real surfaces. 62 and actuators 64. The sensors 62 collectively sense the 

The mouse system 10 may also include an indexing movement of the mouse 12 in the provided degrees of 

function or "indexing mode" which allows the user to 65 freedom and send appropriate signals to the electronic 

redefine the offset between the positions of the mouse 12 in portion of interface 14. Sensor 62a senses movement of link 

the local frame and a user-controlled graphical object, such member 48 about axis A, and sensor 62b senses movement 
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of base member 44 about axis A. These sensed positions tions and other necessary data from RAM 110 and ROM 112 
about axis A allow the determination of the position of as is well known to those skilled in art. In the described 
mouse 12 using known constants such as the lengths of the embodiment, host computer system 18 can receive sensor 
members of linkage 40 and using well-known coordinate data or a sensor signal via a bus 120 from sensors of system 
transformations. 5 10 and other information. Microprocessors 108 can receive 
Sensors 62 are, in the described embodiment, grounded data from bus 120 using I/O electronics 114, and can use I/O 
optical encoders that sense the intermittent blockage of an electronics to control other peripheral devices. Host corn- 
emitted beam. A grounded emitter/detector portion 71 puter system 18 can also output commands to interface 
includes an emitter that emits a beam which is detected by device 104 via bus 120 to cause force feedback for the 
a grounded detector. Amoving encoder disk portion or "arc" 10 interface system 10. 

74 is provided at the end of members 44 and 48 which each Clock 116 is a standard clock crystal or equivalent com- 

block the beam for the respective sensor in predetermined ponent used by host computer 18 to provide timing to 

spatial increments and allows a processor to determine the electrical signals used by host microprocessor 108 and other 

position of the arc 74 and thus the members 44 and 48 by components of the computer system 18. Display device 20 

counting the spatial increments. Also, a velocity of members 15 is described with reference to FIG. 1. Audio output device 

44 and 48 based on the speed of passing encoder marks can 118, such as speakers, can be coupled to host microprocessor 

also be determined. In one embodiment, dedicated electron- 108 via amplifiers, filters, and other circuitry well known to 

ics such as a "haptic accelerator" may determine velocity those skilled in the art. Host processor 108 outputs signals 

and/or acceleration, as disclosed U.S. Pat. No. 5,999,168, to speakers 118 to provide sound out put to the user when an 

and hereby incorporated by reference herein. Sensors 62 are 20 "audio event" occurs during the implementation of the host 

described in greater detail in U.S. Pat. Nos. 6,100,184 and application program. Other types of peripherals can also be 

6,166,723, both incorporated by reference herein. coupled to host processor 108, such as storage devices (hard 

Transducer system 41 also preferably includes actuators disk drive, CD ROM drive, floppy disk drive, etc.), printers, 

64 to transmit forces to mouse 12 in space, i.e., in two (or and other input and output devices. 

more) degrees of freedom of the user object. The bottom 25 Electronic interface 26 is coupled to host computer sys- 
housing plate 65 of actuator 64a is rigidly coupled to ground tern 18 by a bidirectional bus 120. The bi-directional bus 
member 42 (or grounded surface 34) and a moving portion sends signals in either direction between host computer 
of actuator 64a (preferably a coil) is integrated into the base system 18 and the interface device 104. Bus 120 can be a 
member 44. The actuator 64a transmits rotational forces to serial interface bus providing data according to a serial 
base member 44 about axis A. The housing 65 of the 30 communication protocol, a parallel bus using a parallel 
grounded portion of actuator 646 is rigidly coupled to protocol, or other types of buses. An interface port of host 
ground member 42 or ground surface 34 through the computer system 18, such as an RS232 serial interface port, 
grounded housing of actuator 646, and a moving portion connects bus 120 to host computer system 18. In another 
(preferably a coil) of actuator 646 is intergrated into base embodiment, an additional bus can be included to commu- 
member 48, Actuator 646 transmits rotational forces to link 35 nicate between host computer system 18 and interface 
member 48 about axis A. The combination of these rota- device 11. One preferred serial interface bus used in the 
tional forces about axis A allows forces to be transmitted to present invention is the Universal Serial Bus (USB). The 
mouse 12 in all directions in the planar workspace provided USB standard provides a relatively high speed serial inter- 
by linkage 40 through the rotational interaction of the face that can provide force feedback signals in the present 
members of linkage 40. The integration of the coils into the 40 invention with a high degree of realism. USB can also 
base members 44 and 48 is advantageous to the present in source power to drive actuators 64 and other devices of the 
invention and is discussed below. The operation of the present invention. In addition, the USB standard includes 
electromagnetic actuators 64 is described in greater detail in timing data that is encoded along with differential data. 
U.S. Pat. Nos. 6,100,874 and 6,166,723. In other Alternatively, Firewire (also called 1392) can be used as bus 
embodiments, other types of actuators can be used. 45 120; or, the bus can be between an interface card in the host 

FIG. 3 is a block diagram illustrating the electronic computer 18, where the interface card holds components of 

portion of interface 14 and host computer 18 suitable for use device 11 such as microprocessor 130. 

with the present invention. Mouse interface system 10 Electronic interface 26 includes a local microprocessor 

includes a host computer 18, electronic interface 26, 130, local clock 132, local memory 134, sensor interface 

mechanical portion 24, and mouse or other user object 12. 50 136, and actuator interface 138. Interface 26 may also 

Electronic interface 26, mechanical portion 24, and mouse include additional electronic components for communicat- 

12 can also collectively be considered the "force feedback ing via standard protocols on bus 120. In various 

interface device" 11 that is coupled to the host computer. A embodiments, electronic interface 26 can be included in 

similar system is described in detail in U.S. Pat. No. 5,734, mechanical portion 24, in host computer 18, or in its own 

373, which is hereby incorporated by reference herein. 55 separate housing. Different components of interface 26 can 

As explained with reference to FIG. 1, computer 18 is be included in portion 24 or host computer 18 if desired, 

preferably a personal computer, workstation, video game Local microprocessor 130 preferably coupled to bus 120 

console, or other computing or display device. Host com- and may be closely linked to mechanical portion 24 to allow 

puter system 18 commonly includes a host microprocessor quick communication with other components of the inter- 

108, random access memory (RAM) 110, read-only memory 60 face device. Processor 130 is considered "local** to interface 

(ROM) 112, input/output (I/O) electronics 114, a clock 116, device 11, where "local** herein refers to processor 130 being 

a display device 20,and an audio output device 118. Host a separate microprocessor from any processors 108 in host 

microprocessor 108 can include a variety of available micro- computer 18. "Local" also preferably refers to processor 130 

processors from Itenl, AMD, Motorola, or other manufac- being dedicated to force feedback and sensor I/O of the 

turers. Microprocessor 108 can be single microprocessors 65 interface system 10, and being closely coupled to sensors 

chip, or can include multiple primary and/or co-processors. and actuators of the mechanical portion 24, such as within 

Microprocessor 108 preferably retrieves and stores instruc- the housing of or in a housing coupled closely to portion 24. 



06/17/2003, EAST Version: 1.04.0000 



US 6,300, 

13 

Microprocessor 130 can be provided with software instruc- 
tions to wait for commands or requests from computer host 
18, parse/decode the command or request, and handle/ 
control input and output signals according to the command 
or request. In addition, processor 130 preferably operates 5 
independently of host computer 18 by reading sensor signals 
and calculating appropriate forces from those sensor signals, 
time signals, and force processes selected in accordance with 
a host command, and output appropriate control signals to 
the actuators. A suitable microprocessor for use as local 10 
microprocessor 130 includes the 8X930AX by Intel; or 
alternatively the MC68HC711E9 by Motorola or the 
PIC16C74 by Microchip, for example. Microprocessor 130 
can include one microprocessor chip, or multiple processors 
and/or co-processor chips. In other embodiments, micropro- 15 
cessor 130 can include digital signal processor (DSP) func- 
tionality. 

For example, in one host-controlled embodiment that 
utilizes microprocessor 130, host computer 18 can provide 
low-level force commands over bus 120, which micropro- 2 o 
cessor 130 directly transmits to the actuators. In a different 
local control embodiment, host computer system 18 pro- 
vides high level supervisory commands to microprocessor 
130 over bus 120, and microprocessor 130 manages low 
level force control loops to sensors and actuators in accor- 2 s 
dance with the high level commands and independently of 
the host computer 18. In the local control embodiment, the 
microprocessor 130 can process inputted sensor signals to 
determine appropriate output actuator signals by following 
the instructions of a "force process" that may be stored in 30 
local memory and includes calculation instructions, 
formulas, force magnitudes, or other data. The force process 
can command distinct force sensations, such as vibrations, 
textures, jolts, or even simulated interactions between dis- 
played objects. An "enclosure" host command can also be 35 
provided, which causes the microprocessor to define a 
box-like enclosure in a graphical environment, where the 
enclosure has sides characterized by wall and texture forces, 
as described in U.S. Pat. No. 6,100,874. The host can send 
the local processor a spatial layout of objects in the graphical 40 
environment so that the microprocessor has a mapping of 
locations of graphical objects like enclosures and can deter- 
mine interactions with the cursor locally. Force feedback 
used in graphical environments is described in greater detail 
in U.S. Pat. Nos. 6,219,032 and 5,825,308, and co-pending 4S 
patent application no. 08/924,462, all of which are incorpo- 
rated by reference herein. 

Sensor signals used by microprocessor 130 are also 
reported to host computer system 18, which updates a host 
application program and outputs force control signals as 50 
appropriate. For example, if the user moves mouse 12, the 
computer system 18 receives position and/or other signals 
indicating this movement and can move a displayed cursor 
in response. These embodiments are described in greater 
detail in U.S. Pat. No. 5,734,373. In an alternate 55 
embodiment, no local microprocessor 130 is included in 
interface system 10, and host computer 18 directly controls 
and processes all signals to and from the interface 26 and 
mechanical portion 24. 

A local clock 132 can be coupled to the microprocessor 60 
130 to provide timing data, similar to system clock 116 of 
host computer 18; the timing data might be required, for 
example, to compute forces output by actuators 64 (e.g., 
forces dependent on calculated velocities or other time 
dependent factors). In alternate embodiments using the USB 65 
communication interface, timing data for microprocessor 
130 can be retrieved from the USB interface. Local memory 
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134, such as RAM and/or ROM, is preferably coupled to 
microprocessor 130 in interface 26 to store instructions for 
microprocessor 130 and store temporary and other data. 
Microprocessor 130 may also store calibration parameters in 
a local memory 134 such as an EEPROM. As described 
above, link or member lengths or manufacturing variations 
and/or variations in coil winding or magnet strength can be 
stored. Memory 134 may be used to store the state of the 
force feedback device, including a reference position, cur- 
rent control mode or configuration, etc. 

Sensor interface 136 may optionally be included in elec- 
tronic interface 26 to convert sensor signals to signals that 
can be interpreted by the microprocessor 130 and/or host 
computer system 18. For example, sensor interface 136 can 
receive signals from a digital sensor such as an encoder and 
convert the signals into a digital binary number representing 
the position of a member of mechanical apparatus 14. An 
analog to digital converter (ADC) in sensor interface 136 
can convert a received analog signal to a digital signal for 
microprocessor 130 and/or host computer 18. Such circuits, 
or equivalent circuits, are well known to those skilled in the 
art. Alternately, microprocessor 130 or host 18 can perform 
these interface functions. Other types of interface circuitry 
136 can also be used, e.g., the electronic interface described 
in U.S. Pat. No. 5,576,727, which is hereby incorporated by 
reference herein. 

Actuator interface 138 can be optionally connected 
between the actuators 64 and microprocessor 130. Interface 
138 converts signals from microprocessor 130 into signals 
appropriate to drive the actuators. Interface 138 can include 
power amplifiers, switches, digital to analog controllers 
(DACs), and other components. Such interfaces are well 
known to those skilled in the art. In alternate embodiments, 
interface 138 circuitry can be provided within microproces- 
sor 130 or in the actuators. 

In the described embodiment, power is supplied to the 
actuators 64 and any other components (as required) by the 
USB. Since the electromagnetic actuators of the described 
embodiment have a limited physical range and need only 
output, for example, about 3 ounces of force to create 
realistic force sensations on the user, very little power is 
needed. For example, one way to draw additional power 
from the USB is to configure device 11 to appear as more 
than one peripheral to host computer 18; for example, each 
provided degree of freedom of mouse 12 can be configured 
as a different peripheral and receive its own allocation of 
power. Alternatively, power from the USB can be stored and 
regulated by device 11 and thus used when needed to drive 
actuators 64. For example, power can be stored over time 
and then immediately dissipated to provide a jolt force to the 
user object 12. A battery or a capacitor circuit, for example, 
can store energy and discharge or dissipate the energy when 
power is required by the system and/or when enough power 
has been stored. Alternatively, a power supply 140 can 
optionally be coupled to actuator interface 138 and/or actua- 
tors 64 to provide electrical power. The power storage 
embodiment described above, using a battery or capacitor 
circuit, can also be used in non-USB embodiments to allow 
a smaller power supply 140 to be used 

Mechanical portion 24 is coupled to electronic portion 26 
and preferably includes sensors 62, actuators 64, and linkage 
40. These components are described in detail above. Sensors 
62 sense the position, motion, and/or other characteristics of 
mouse 12 along one or more degrees of freedom and provide 
signals to microprocessor 130 including information repre- 
sentative of those characteristics. Typically, a sensor 62 is 
provided for each degree of freedom along which mouse 12 
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can be moved, or, a single compound sensor can be used for In some embodiments of interface system 10, multiple 

multiple degrees of freedom. Example of sensors suitable mechanical apparatuses 102 and/or electronic interfaces 100 

for embodiments described herein are rotary optical can be coupled to a single host computer system 18 through 

encoders, as described above, linear optical encoders, analog bus 120 (or multiple buses 120) so that multiple users can 

sensors such as potentiometers, or non-contact sensors, such 5 simultaneously interface with the host application program 

as Hall effect magnetic sensors or an optical sensor such as (in a mu i t i- p iayer game or simulation, for example). In 

a lateral effect photo diode having an emitter/detector pair. addition, multiple players can interact in the host application 

In addition, velocity sensors e.g., tachometers) and/or program with mu i up l e interface systems 10 using networked 

acce eration sensors (e.g., accelerometers) for measuring host ters 18 „ fc well t0 ^ skilled m the 

acceeration can be used. Rirthermore, either relative or 1Q art . ^ me mterface dev ice 104 can be coupled to multiple 

absolute sensors can be employed. host computers; for example> a local host compuler caD 

Actuators 64 transmit forces to mouse 12 in one or more display images based on data received from a remote host 
directions along one or more degrees of freedom in response compute r coupled to the local host through a network, 
to signals output by microprocessor 130 and/or host com- 
puter 18, i.e., they are "computer controlled." Typically, an 15 Architecture of Host Computer 
actuator 64 is provided for each degree of freedom along The host computer 18 interacts with interface device 11, 
which forces are desired to be transmitted. Actuators 64 can in the present invention, using a number of different levels 
be the electromagnetic actuators described above, or can be of controllers. These controllers are preferably implemented 
other active actuators, such as linear current control motors, in software, e.g. program instructions or code, and such is 
stepper motors, pneumatic/hydraulic active actuators, a 20 me embodiment described herein; however, all or part of the 
torquer (motor with limited angular range); or passive controllers may also be implemented in hardware, where the 
actuators such as magnetic particle brakes, friction brakes, conversion of functionality of the software to hardware is 
or pneumatic/hydraulic passive actuators, passive damper well known to those skilled in the art. 
elements, an electrorheological fluid actuator, a magne- The architecture described herein is oriented towards 
torheobgical fluid actuator In addition, in voice coil ^ providing force feedback functionality to a host computer 
embodiments, multiple wire coils can be provided, where connected to a force feedback interface device, where mul- 
some of the coils can be used to provide back EMF and tiple applications may be running simultaneously on the host 
damping forces. In some embodiments, all or some of computer. Each application program may have its own set of 
sensors 62 and actuators 64 can be included together as a f orce sensations to output to the device; since the device 
sensor/actuator pair transducer. 30 cannot implement all force sensations at any one time due to 

Mechanism 40 is preferably the five-member linkage 40 cost and hardware constraints, the forces commanded by 

described above, but can also be one of several types of each application must be organized by the architecture to 

mechanisms. For example, mechanisms disclosed in take these limitations into account. 

co-pending patent applications Ser. Nos. 08/664, 086, FIG. 4 is a block diagram of a preferred architecture for 

08/709,012, and U.S. Pat. Nos. 5,731,804; 5,767,839; 5,721, 35 me nos t computer to communicate with and control a force 

566; 5,805,140; 5,691,898; and 5,828,197, all incorporated feedback interface device 11 with multiple application pro- 

by reference herein, can be included. Mouse 12 can alter- grams r unnin g on the host computer. Application programs 

natively be a puck, joystick, or other device or article 202 and 204 are concurrently running on the host computer 

coupled to linkage 40, as described above. 18 . i n me most typ^ implementation, one of the applica- 

Other input devices 141 can optionally be included in 40 tion programs is actively running in an operating system as 
system 10 and send input signals to microprocessor 130 the "active" application program that displays one or more 
and/or host computer 18. Such input devices can include active windows in the GUI (also known as the application 
buttons, such as buttons 15 on mouse 12, used to supplement program that is "in focus" or which has "keyboard focus"), 
the input from the user to a GUI, game, simulation, etc. Also, The active window is typically the topmost displayed win- 
dials, switches, voice recognition hardware (with software 45 dow in which input is provided by the user using the 
implemented by host 18), or other input mechanisms can be mouse -controlled cursor, a keyboard, or other peripheral, 
used. The other applications are "inactive*' in that they are not 

Safety or "deadman" switch 150 is preferably included in receiving input from the user (although they may have a 
interface device to provide a mechanism to allow a user to window displayed in the GUI which can be updated on the 
deactivate actuators 64 for safety reasons. In the preferred 50 screen). The inactive applications may also receive input or 
embodiment, the user must continually close safety switch send output. For example, the Windows™ operating system 
150 during manipulation of mouse 12 to activate the actua- from Microsoft Corp. provides a multitasking or pseudo- 
tors 64. If, at any time, the safety switch is deactivated multitasking environment in which programs run simulta- 
(opened) the actuators are deactivated while the safety neously; other operating systems such as Unix also offer 
switch is open. For example, one embodiment of safety 55 multitasking. For example, a word processor may be the 
switch is a capacitive contact sensor that detects when the active application to receive input from the keyboard and 
user is contacting mouse 12. Alternatively, a mechanical display input characters in a displayed active window on 
switch, electrostatic contact switch, or optical switch located display screen 20, while a inactive communication program 
on mouse 12 or on a convenient surface of a housing 21 can may be receiving data over a network and saving the data on 
be used. A z-axis force sensor, pieao electric sensors, force 60 a storage device, or sending data out to be printed. Or, a 
sensitive resistors, or strain gauges can also be used. The inactive drawing program may be displaying a graphical 
state of the safety switch can be sent to the microprocessor image in a window that is not currently active, while the user 
130 or host 18. In one embodiment, mouse 12 includes a inputs text into the active word processor window. When the 
hand weight safety switch. The safety switch preferably user moves me cursor over an inactive window and provides 
deactivates any generated forces on the mouse when the 65 a command gesture such as clicking a button on a mouse, the 
mouse is not in use and/or when the user desires to deacti- inactive window becomes the active window and the former 
vate output forces. active window becomes inactive. The active application is 
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also known as the "fo re ground" application, in the sense that tions 244. For example, a user can specify positive or 
its force sensations are being implemented by the force negative damping (where negative damping feels like mov- 
feedback device, as described below. ing on ice) or unidirectional or bidirectional dampers. The 
A master application 206 also is running on host computer user can specify the spacing or orientation of snap points on 
18 and is also known as the "background" force feedback 5 a grid, the envelope parameters for a vibration, or the 
application. This application is preferably a general purpose parameters for a snap force, e.g. the snap force is defined as 
program that always runs inactively in the operating system an enclosure and the user can customize the enclosure to turn 
and whose set of commanded forces are always available to off forces on one waU> turn on inner walls, etc. Such options 
be output and controlled on the interface device 11 and/or can be similar t0 those described m us . Pat . Nos . 6,147,674 
other devices. For example, a preferred embodiment of the 10 and 6 ,169,540 or those features provided in the force design- 
master application is a "desktop" control panel for force ^ application lJPcm Studio from T mmer sion Human 
feedback. An example of such a program is illustrated in Inte rface Corporation. Other application programs can also 
FIG. 5. A "mouse properties" dialog box 240 can be dis- be ^ M the background or master application program 
played which allows the user to specify force sensations 244 206 

that are assigned to specified object types 242 in the dis- 1C n ~ . . , , A ^ M ^ A 

1 a u * 1 * * w 1 • * _r 15 Referring back to FIG. 4, application programs 202, 204, 

played graphical environment, e.g. a graphical user interface , „^ 6 . ' ' , , * . 

(GUI). For example, the assigned I for« sensation 244 can be i Dd 206 <^mun,cate ™* » ,° *«» ^ck Application 

a vibration force having a specified frequency, magnitude, «g Interface ("API ) 208 which is resident in the 

j . • i « » ■ i* * -a j *• j host computer s memory and which allows a given appli- 

duration, etc.; a single "pop or jolt at a specified tune and * , ; . ( ., , i Fj • j 

j 4 . i >» *if» ■ ii ■ i4 *u 4 • * u a cation program to communicate with lower level drivers and 

duration; a tick that is a small jolt that is output based on « n iL / °* r ,, , - „ t „ T . 

* r 4 . . i - * i w other force feedback functions. For example, in the Win- 
movement of the user object/cursor or at regular intervals , .a™, i j . « 

during an activity such as scrolling; a damping force con- °P craUn S s y stcm ' ^ s m «™ly ^ /° aUow 

dition that causes a resistance to mouse motion depending ^developer of an apphcaUon program to call functions at a 

onthevelodtyofmeuserobjectl2iniudegreesoffreedom hl S h lev K cl f 0 ' »f ™ th * c application program and not 

f . „ irvnr L k«c# f„ mo n „ ->n „ r - n tUni worry about the low level details of actually implementing 

or the cursor in the host trame on screen 20; or a spring that 2 $ me fjj^ctioQ 

resists motion of the user object based on distance to an 

origin of the spring. Other types of forces are possible as TheAPIof the present invention includes a set of software 

well, and are described in greater detail in U.S. Pat. Nos. "objects" that can be called to command a force feedback 

5,734373 and 6,169,540, incorporated by reference herein. interface device 11. Objects are a set of instructions and/or 

The force sensations specified by the user for a user 30 data which can be called by a pointer and/or an identifier and 

interface object type in dialog box 240 will be output by the palters to implement a specified function. For example, 

force feedback device by default, unless a different fore- mre f t > r P es ? f are Prided in one preferred API 

ground application program deactivates the force sensations implementation: mterf ace objects, device objects, and effect 

or replaces a force sensation with its own. For example, the Each of these objects makes use of one or more 

user can specify that menu objects 242a in a GUI have a 35 f< ? rce feedback device dnvers which are implemented as 

snap force 244a associated with them by moving the slider ooj^ 111 ™c API 208. 

246 to the right, where the slider scale specifies the magni- Interface objects represent the API at the highest level. An 

rude of the snap force. Thus, when the user moves the cursor application program (which is referred to as a "client" of the 

in the GUI over a menu item in a menu during normal A** 1 ) ^ CTeate an interface object to access lower level 

operation, the user will feel a snap force, i.e., a force that 40 objects and code that implement force feedback device 

draws the cursor/user object toward the center of the menu functionality. For example, the interface object allows an 

item to allow easier menu selection. This snap force is application program to enumerate and receive information 

preferably applied to all menus of all running application about individual devices and create lower level objects for 

programs in the GUI, since it is a background force effect. cacD device used by the application program. 

If the foreground application program has its own force 45 Device objects each represent a physical force feedback 

sensations which define that application's menus to have a device (or other compatible device or peripheral) in com- 

jolt instead of a snap, then the jolt will be superimposed on munication with the host computer 18. For example, the 

the snap unless the active application program deactivates force feedback mouse device 11 would be represented as a 

the background force(s). In general, a particular active single device object. The device objects access the set of 

application program can only command forces to objects in 50 force feedback routines to receive information about an 

its own windows, e.g., that application's own menus, associated physical device, set the properties of the physical 

scrollbars, icons, window borders, etc. device, register event handlers (if events are implemented on 

A user can specify multiple background force sensations ^ e host), and to create effect objects. 

244 for each user interface object 242. This allows the Effect objects each represent a force feedback effect 

multiple force sensations to be superimposed on each other, 55 defined by the application program to be output one or more 

unless the application program overrides one or more of the times on a force feedback device. The effect objects access 

superimposed forces. Thus, a user can assign a damper force the set of force feedback routines to download force effects 

sensation and a "ticks" force sensation to scrollbars in box to the force feedback device, start and stop the output of 

240, and all scrollbars will have these forces associated with effects by the force feedback device, and modify the param- 

them unless overridden by the foreground application pro- 50 eters °f me effects. Event objects can also be created to 

gram. define events, as described in greater detail below. 

Other controls in box 240 include a device gain 248 to set A force "effect, " as referred to herein, is a single defined 

the percentage of maximum magnitude for all the forces for force or series of forces that may be commanded within the 

items 242; and selections 249 to allow force feedback API. The effect typically has a name to identify it and one 

functionality on the host computer. Advanced button 250, 65 or more parameters to modify or customize the force as 

when selected, preferably displays an additional window of necessary. For example, several types of force effects have 

options with which the user can customize the force sensa- been defined, including vibrations, enclosures, grids, 
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textures, walls, dampers, snap sensations, vibrations, circles, to command a force feedback device, the application will 
ellipses, etc. For example, a vibration force effect may have query for the API 208. Once communication is established, 
parameters of duration, frequency, magnitude, and direction. the API will contact the context driver 210 to create an entry 
The force sensations 244 defined in dialog box 244 can be location for a context set for the initiated application pro- 
force effects; however, more generally, force sensations can 5 gram. 

be made up of one or more force effects. Force effects, in The context driver 210 receives individual effects and 

turn, are made up of one or more basic force prototypes, events as they are created by the application program using 

such as springs, dampers, and vector forces. API 208 and stores the effects and events in a context list 

In the preferred embodiment, an application program 212, storing each context in a different storage location in 

client interacts with the API 206 by first receiving a pointer 10 the host's memory or on some other type of storage device, 

to a resident force feedback interface; for example, a pre- An active or inactive application program can create a 

ferred interface includes procedures provided in the Com- context and have it stored, but only the active application's 

ponent Object Model (COM) interface of Microsoft context will be sent to the force feedback device. The 

Windows, an object oriented interface. Using the force context driver 210 can examine an identifier in a created 

feedback interface, the application program enumerates the „ effect or event to determine which application program is 

force feedback devices on the computer system 18. Hie associated with it and thus store the effect or event m the 

application program selects a desired one of the force P ro P er mcmofy 2^ dr J vcr f h te P 01 f tc f 0 

r jl 1 j • j * ~i * * *. j *+u the contexts and to the effects and events m the contexts to 

feedback devices and creates a device object associated with them ^ ^ caQ be create(j bcfore 

that device. Using the force feedback interface, the apph- forccs afe commandcd) or me effect can be created and then 

cation then acquires the device, sets the device up with setup 20 ^^j^y commanded to be output to the force feedback 

and default parameters, and creates effect objects and event device Each context also preferably includes an entry into 

objects at times designated by the developer of the applica- a device state structure. This entry governs the "gain" or 

tion. At appropriate times, the application program will forcc j evel fo r ^1 effects j n ^ c con text. For example, all the 

command the start, stop, or pause of the playback of force forces in the context can be cut to 50% of full magnitude by 

effects by accessing the appropriate interface instructions 25 storing a value of 50 in the device state structure. One of the 

associated with the desired effect. The API indicates to the contexts stored in list 214 is a primary context 216, which 

context driver 210 to create a "context" (i.e. "effect set") for is the list of effects and events used by the master application 

an application program, and sends effects and events to be 206 or "background" application. 

associated with that context. A "context" is a group or set of At a later time, after the context driver has stored the 
effects and events that are associated with a particular 30 contexts in list 212, an application program may send a 
application program. command to the API to output or "start" a particular force 

Context driver 210 receives contexts 222 and device sensation. The API checks whether the application program 
manipulation data 223 from the API 208. The context driver is active or in the background (primary); if not, the API 
is provided at a level below the API to organize contexts for ignores the command. Alternatively, the commands from an 
applications 202 and 204 running on the host computer. In 35 inactive foreground application can be stored and then sent 
the preferred embodiment, the effects and events in a context to the device when the application becomes active. If the 
are not known to the application program itself; rather, the application is active or background, the API sends the start 
context driver 210 creates a context internally for an appli- information to the context driver 210 indicating the appli- 
cation. Thus, an application commands that a particular cation program that commanded the force and the particular 
force sensation be output in response to different interactions 40 force effects to be commanded. The context driver 210 then 
or occurrences, e.g., an interaction of a cursor with an object associates the commanding application program with a 
or region, or the output of a force based on other criteria context 214 in list 212 and sends the effects from the context 
(time, received data, random event, etc.). The API sends the to the force feedback device (if not previously sent). For 
commanded effect(s) to the context driver 210, and the example, if a context for a particular application program 
context driver stores the effects in the created context for that 4S includes a spring effect, a damper effect, and a vibration 
application program. effect, and the application program commands the vibration 

Since each application may have a completely different to be output, then the context driver selects the vibration 
set of force effects to output, each context each must be effects to be output to the device. The data describing this 
associated with its particular application program. In the effect is then output by the context driver 210. Similarly, the 
preferred embodiment, there are two types of contexts: 50 application program can send a command to stop particular 
foreground contexts and background or primary contexts. A force effects, to pause the effects, to get the status informa- 
foreground context is associated with the application pro- tion of an effect, or to destroy an effect, 
gram 202 or 204 that is active in the operating system. A A context is therefore only allowed to exert forces with 
background context includes the effects and events for the the force feedback device when that context is active, i.e., is 
master application program 206, e.g., the effects and events 55 associated with an active application program or the back- 
necessary to implement those force sensations selected by ground application. In the described embodiment, only one 
the user in the dialog box 240 of FIG. 5 to be associated with foreground context can be active at any given time. Any 
particular GUI objects. In addition, a "global context" can be number of background contexts may be simultaneously 
provided, which includes common effects almost always active; however, there may be a device limit on the number 
used by application programs (e.g. a pop force) and which 50 of background contexts that may be created. For example, 
can automatically be downloaded to the force feedback the mouse device U may only allow one background 
device. Effects in the global context need not be stored in context to be created at any one time, which is the preferred 
individual contexts for particular application programs and embodiment. Thus, if an inactive (not in focus) foreground 
need not be sent to the force feedback device, thus saving application program commands a force to be output, the API 
memory on the device. 65 will ignore the command after determining that the com- 

When an application program is first executed by the host manding application is not active (or, the command will only 
computer and loaded into memory, if the application is able be sent to the device when the application becomes active). 
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If the active application program becomes inactive (i.e. stored, such as an enable or disable for all forces, so that if 

loses foreground or is removed from the host's memory) and all forces are disabled, no forces will be output by the 

a different application becomes active, then the API indi- device. An overall gain can also be set to limit all output 

cates this to the context driver 210, which then deactivates force magnitudes to a desired level or percentage of maxi- 

the context associated with that application program and 5 mum output. 

loads the effects from the new active context to the force The translation layer outputs device messages 225 to the 

feedback device 11. Likewise, when the original application device 11 by way of the next layer. Messages may include 

program again becomes active, the API tells the context f orce effects to be output and/or any other information such 

driver to activate the associated context and load the appro- as device identification numbers or instructions from the 

priate effects to the force feedback device. 10 context driver for an effect (start, stop, pause, reset, etc.) The 

Device manipulation data 223 is also received by context translation layer outputs messages 225 in a form the device 

driver 210. Data 223 is used to set a global device state on 11 can understand. 

the force feedback device, as described below, and this Device communication driver 226 communicates directly 

information is passed unmodified to the translation layer. with the force feedback device. Driver 226 receives the 

Translation layer 218 manages the sending of effects to 15 device messages 225 from translation layer 218 and directly 

the device 11, receives information from the device to the transmits the messages to the force feedback device over bus 

host, and maintains a representation of the memory of device 120, e.g. a USB. The driver is implemented, in the preferred 

11. Layer 218 receives an individual effect 219 for the active embodiment, as a standard driver to communicate over such 

(or background) application program and device manipula- a serial port of host computer 18. Other types of drivers and 

tion data 223 sent by the context driver 210. The translation 20 communication interfaces can be used in other embodi- 

layer receives the effect from an context list 220 of indi- ments. 
vidual effects 222 (list 220 represents a context 214). A 

different context list 220 is provided in each context 214 of Host Command Structures 

list 212. Each effect 222 in list 220 is a force that is to be host computer 18 and API 208 handle a variety of 

output on the mouse 12 by the force feedback device 11. processes controlling force feedback device 11. The pre- 

When the effects are to be sent to the device 11 when ferred functionality of much of the application programs and 

commanded by the application, they are selected and copies ^ described in greater detail in U.S. Pat. Nos. 6,147,674 

are output to the device. Preferably, each effect is output by an a 6,078,308 and co-pending patent application No. 

the translation layer as soon as possible, in the order of 08/924,462. Different effects and events of the present 

receiving the effects. Each effect stored in list 220 as invention used in force feedback systems are described 

examined by the translation layer is available on force below, 
feedback device 11, i.e., the local microprocessor should 

recognize the effect and be able to output the effect either Effects 

«^ d u iate ,5 , u ^W ^ enOTndid0m ^ C , , r' te •? eS ? e0ftb f!? , 35 Effects are standardized forces that are determined 

220 should be the same or smaller than the available accordin to , predefined characterization, which may 

memory for such a list in the force feedback device so that force al rithms> stored force magnitudes , functions 

the translation layer knows when the memory of the force ofspaceand/ortime>ahistoryof stored motion values of the 

feedback device is full If full, the translation layer can delay ffl Qr other Qr arameters . ^ effect may ^ based 

the output of additional effects until enough memory space ^ Qn time ^ be mdependent of me of me mouse ^ 

is aval a or may be spatially determined based on mouse position, 

When an active application becomes inactive, the trans- velocity, acceleration, or any combination of these, 

lation layer is instructed by the context driver 210 to Preferably, the device 11 includes a number of effects 

"unload" the effects of the context of the previous active standardized in its memory which it can implement if the 

application from the force feedback device, receives the 45 e ff ec ts are within the active or background context. When 

effects from the now active application and loads those me device 11 is determining output forces based on effects, 

effects to the force feedback device 11 (the effects for the me microprocessor 130 checks if the effect is active, calcu- 

background (primary) application are preferably never lates the raw contribution to the output force from the effect, 

unloaded). applies an envelope scaling (detailed in U.S. Pat. No. 

The translation layer also preferably handles events. For 50 5,959,613, incorporated by reference herein), and sums the 
example, when a screen position is received from the device scaled contribution to the total sum of forces contributed to 
1, the translation layer can check whether any of the by all effects currently being output. In determining the total 
conditionsAriggers of the active events are met If so, a sum, the device preferably combines all constant forces 
message is sent which eventually reaches the associated (e.g., conditions and time-based forces) and limits the con- 
active or background application program, as described in 55 stant force sum to a predetermined magnitude, then corn- 
greater detail below. In alternate embodiments, the micro- bines all dynamic forces and limits the dynamic force sum 
processor 130 on device 11 can check for events and send to a predetermined magnitude. Dynamic forces are detailed 
event notifications through translation layer 218 up to the in U.S. Pat. No. 6,147,674. The two sums are then added 
application program. However, this can be undesirable in together and the total force sum is output by the actuators of 
some embodiments since the event notification is provided 50 the device 11. Alternatively, all forces can be treated the 
at a much slower rate than the force control loop of the same and summed together. 

microprocessor 130. FIGS. 6a and 6b illustrate one type of force effect 

The translation layer also can store a device state 224 in provided by the host in a graphical environment such as a 

memory. Device manipulation data 223 from the active GUI and known as an "enclosure/* Basic features of enclo- 

application determines the device state. This is the state that 65 sures are disclosed in U.S. Pat. No. 6,078,308, incorporated 

the active application program wishes to impose on the by reference herein. An enclosure is at its most basic level 

device, if any. For example, an overall condition can be a rectangular object provided in a GUI which may be 
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assigned with different forces for each of its sides. The 
enclosure is a generic type of force feedback object that can 
be used for a variety of different graphical objects, including 
buttons, windows, sliders, scrollbars, menus, links on a web 
page, or grid points in a drawing program. 

FIG. 6a illustrates the use of an enclosure for a window 
in a GUI. The window 260 is displayed on the display device 
of the host computer. The enclosure has eight walls 262 
associated with it invisible to the visual perception of the 
user, where four walls are provided as an inner rectangle 264 
and four walls are provided as an outer rectangle 266. Each 
wall 262 can be separately defined as either providing a 
force pulling the user object toward the border of window 
260 (attractive force), or pushing the user object away from 
the border of window 260 (resistive force). The outer 
rectangle 266 walls affect the user object when the user 
object is outside the window 260 between the rectangle 266 
and the window 260 border, and the inner rectangle 264 
walls affect the user object when it is inside the window 260 
between the rectangle 264 and the window border. In the 
preferred embodiment, a window 260 is associated with a 
force enclosure by defining both outer rectangle 266 and 
inner rectangle 264 to have an attractive force, so that the 
user object and cursor tend to be attracted to the window 
border when within the appropriate range of the window. 
This, for example, allows the user to acquire the border and 
adjust the size of the window easily. Alternatively, the outer 
rectangle 266 can be defined as a resistive force (such as a 
spring) that requires the user to overcome the resistance to 
allow the user object entry into the window (and the rect- 
angle 266 can also be defined as a click surface to cause a 
function to be applied when gaining entry). When the 
cursorluser object gains entry to an enclosure, an "event" is 
preferably indicated to the application program, as explained 
above. In an alternative embodiment, enclosures can simi- 
larly defined for circles, ellipses, or other shapes, both 
regular and irregular. 

The enclosure object also preferably has a number of 
other parameters that further define the enclosure. For 
example, parameters may be specified separately for each 
wall to define such characteristics as the magnitude of the 
resistive or attractive forces, which of the eight walls are 
assigned forces, which walls provide clipping, the saturation 
level of the force applied in association with a wall, and the 
interior force, which can be specified as any force effect such 
as a damping, texture, or vibration force, that is applied 
while the cursor is inside the enclosure. 

FIG. 6b illustrates another embodiment of an enclosure. A 
button 270 can be a graphical button, a menu item, or an icon 
in a GUI. The outer rectangle 266 in this embodiment has 
been "turned off" so that no forces are associated with it. The 
inner rectangle 264 has been defined as a very small rect- 
angle at the center of the button 270. Resistive forces are 
associated with the inner rectangular walls such that, once 
the user object has entered the enclosure, a force biases the 
user object toward the center of the button 270. This allows 
the user to target or "acquire" the button more easily with the 
cursor and reduces overshooting the button by accident by 
moving the cursor too far past the button. 

Enclosures are quite useful for adding forces to web links 
on a web page viewed in a web browser, such as Netscape 
Communicator by Netscape Communications or Internet 
Explorer by Microsoft. The inner wall forces of FIG. 6b can 
be used on a web link displayed on a web page to draw the 
cursor/user object onto the link, thus allowing a user to more 
easily determine which images are links on the page. Other 
types of forces can also serve this function. Also, an "inside 
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condition" such as a texture can be provided for the web link 
to cause some links to feel rough, some smooth, some 
bumpy, etc. 

An example of an enclosure effect structure is provided 
5 below: 

ENCLOSURE 
typedef struct { 
DWORD dwSize; 
RECT rectOutside; 
10 DWORD dwXWidth; 
DWORD dwYWidth; 
DWORD dwXStifmess; 
DWORD dwYStiffhess; 
5 DWORD dwStiffiaessMask; 
DWORD dwClippingMask; 
DWORD dwXSaturation; 
DWORD dwYSaturation; 
LP 13 CONDITION plnsideCondition 
20 } ENCLOSURE; 

dwSize: Size of the structure in bytes. 
rectOutside: Outside borders of the enclosure, in screen 
coordinates. When the mouse position is outside of rect- 
angle defined by rectOutside, there are no forces exerted 
25 on the device from this enclosure. 

dwXWidth, dwYWidth: Width W of the region between a 
wall and the object border. In one implementation, the 
width of the inner wall and the outer wall regions is the 
same. Alternatively, they can be defined separately. This 
30 parameter can alternatively be specified as a deadband. 
dwXStiffness: Stiffness of the left and right walls. Defines 
the amount of horizontal resistance (force) felt by the user 
when attempting to break through a vertical wall. This 
value amounts to the negative and positive coefficient of 
35 horizontal spring conditions that are applied when the 
mouse is in the left and right walls, respectively. 
dwYStiffness: Stiffness of the top and bottom walls. Defines 
the amount of vertical resistance (force) felt by the user 
when attempting to break through a horizontal wall. 
40 dwStiffnessMask: Mask specifying when resistance should 
be applied. Can be one or more of the following values: 
NONE: No resistance is felt on any wall. 
INBOUND TOP WALL, INBOUNDLEFTWALL, etc.: 
Resistance is felt when entering the enclosure through 
45 the top, bottom, left, or right wall, as specified. 

OUTBOUNDTOPWALL, OUTBOUNDLEFTWALL, 
etc.: Resistance is felt when exiting the enclosure 
through the top, bottom, left, or right wall, as specified. 
5Q INBOUNDANYWALL, OUTBOUNDANYWALL: 
Resistance is felt when entering or exiting, respectively, 
the enclosure through any wall. 
ANYWALL: Resistance is felt when entering or exiting 
the enclosure through any wall. 
55 dwClippingMask: Mask specifying when mouse cursor clip- 
ping should be applied. Can be one or more of the 
following values: 

NONE: Clipping is not applied to any wall 
INBOUNDTOPWALL, INBOUNDLEFTWALL, etc.: 
60 Clipping is applied when entering the enclosure 
through the top, bottom, left, or right wall, respectively. 
OUTBOUNDTOPWALL, OUTBOUNDLEFTWALL, 
etc.: Clipping is applied when exiting the enclosure 
through the top, bottom, left, or right wall, respectively. 
65 INBOUNDANYWALL, OUTBOUNDANYWALL: 
Clipping is applied when entering or exiting, 
respectively, the enclosure through any wall 
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ANYWALL: Dipping is applied when entering or exiting associated force to attract the cursorluser object to the 

the enclosure through any wall. border Buttons 277 are also included in window 272 which 

dwXSaturation, dwYSaturation: Saturation level of spring are near the border 275 and which may be selected by the 

effect felt by the user when attempting to break through a user. Buttons 277 have their own enclosures defined in the 

vertical or horizontal wall, respectively. 5 interior of the buttons. The problem is that the forces from 

plnsideCondition: Points to a condition that is to be in effect enclosure 274 and the forces from the button enclosures 

when the user mouse is in the deadband area. For conflict. A user may not be able to select the button easily 

example, can specify a spring condition, damper J™ t0 te bein S stuck bttwcea enclosure 

condition, and/or texture condition inside the enclosure. J?f s ; Ul , . , ~ A t 4 , 

... . . , oiTi* Preferably, an enclosure 276 can be defined to control the 

Tins is useful m graphical environments such as GUI s 10 ^ q{ cndosures Fof le> mc cndosurc 276 

and web pages/web browsers. occupies a region around buttons 277 and is preferably not 

In another example of an enclosure a circular or elliptical yisib [ e tQ Eaclosure 276 turns off the forces of 

shape is defined. For example, an elliptical enclosure can enclosure 274 inside its borders and leaves the snap forces 

include only one wall that is curved all the way around the of mc buttons 277 on. This allows the user to feel the snap 

ellipse. The inner and outer walls can still be provided, as 15 f orces 0 f me buttons without conflict from the forces of the 

well as the rest of the other parameters. The dimensions of window enclosure 274. Alternatively, the forces of enclosure 

the ellipselcircle can be provided as X and Y widths. 274 can be reduced inside enclosure 276 to an extent so that 

Alternatively, an eccentricity parameter can define the the user is not encumbered by competing forces. A similar 

eccentricity of the ellipse. type of enclosure can be user on sliders, where the moveable 

One problem with enclosures is that often the forces of the 20 "thumb" of the slider can be provided with an enclosure 

enclosure are desirable at one point in time but not at a similar to enclosure 276 surrounding it. The enclosure 

different point in time. For example, a button having an surrounding the slider turns off or reduces forces from other 

enclosure has a attractive force associated with the enclosure enclosures and allows the snap attractive force of the thumb 

to assist the user in acquiring the button with the cursor. attract the user object/cursor to the thumb. Events can be 

However, once the cursor is positioned inside the enclosure, 25 to determine when the cursor enters enclosure 276. 

the user may want to be able to freely move the cursor again *}? 7 Ulustrates a different effect useful in force feedback 

without enclosure wall forces, since the target button has appkeatror* known as a texture. A texture is a condition 

already been acquired. Thus, in one embodiment, the forces smi1 ^ to which causes the mouse to feel as if it 

associated with the enclosure are turned off once the cursor were * avehn g over a ^ nes ^umps. FIG. 7 illustrates a 

is moved inside the enclosure. Alternatively, the force mag- 30 ^tance Vi V force graph in which a number of bumps are 

nitudes can be weakened or adjusted to a lower level, e.g. >/ 3 l efined ^ ^efficient Q a period T, and a deadband DB. 

the normal magnitude, to allow the cursor to easily exit the **** * the positive distance region are felt when the 

enclosure. Once the cursor has exited, the forces can be mouse 12 » moved m a P° slUv ^ f e< f> n ' f d bum ^ 

turned back on. In an alternate embodiment, the forces fCg T m . fcl * when the mouse u 12 * 

associated with the enclosure can be turned off or weakened 35 m n ^ atlve f 1 "?*} 0 ^ ™ CfC f 1S 3 j**™**. 

for a predetermined period of time to allow the user to move (efficient) and period of the bumps for the positive and 

the cursor out of the enclosure without hindrance, and then negaUve cbrecnons along a particular axis of the mouse 12 

meforcescanbeautomaucallymmedbackononcemetime » mat d f e ^ nt textu ^ b * fel * ^pendiag on the 

has expired. In yet a different embodiment, the forces are directl ° n of t^ mouse. Direcuona^ [textures can be used, for 

turned off only when the cursor has not moved for a 40 exam P le > to * ch £ v * a ^Ivet-hke ^ deadband value 

predetermined period of time; this indicates that the user has ^an apply for both directions. Textures are defined as a 

acquired the desired target. In still a further embodiment, the 000(1111011 ™ d can mclude Parameters such as: 

forces can be turned off when a command gesture, such as Posilive Coefficient, Negative Coefficient: Height 

a button 15 activation, is detected by the host or micropro- (magnitude) of bumps when travelling along the axis in 

cessor 130. These various embodiments can also be com- 45 me P 05 ^ or negative direction, respectively, 

bined in other embodiments as desired. The host or micro- Positive Saturation, Negative Saturation: Period, in 

processor can determine when the enclosure has been P« el s, of bumps (bump width plus deadband width) 

entered and exited by the cursor through the use of events, when travelling along the axis in the positive or nega- 

explained below. uve direction, respectively. 

Enclosures can also be defined to compensate for user 50 DeadBand: Percentage of period which is not occupied by 

controlling peculiarities. For example, it has been found that a bump. 

up-down movement of the mouse 12 for most users is much FIG. 8 illustrates a different force feedback effect useful 

easier that left-to-right movement. An enclosure having in force feedback applications known as a grid. A grid is a 

equal strength forces on all walls will feel as if the left and condition that creates a 2 -dimensional array of snap points 

right walls have stronger forces. Thus, the upper and lower 55 280, which are forces attracting the user object to the point 

walls can be defined with stronger forces, e.g., two times the when the user object moves over the point. Grids can be 

magnitude of the left and right wall forces, to compensate stand-alone conditions, but they can also be useful when 

for this effect. Such an enclosure typically feels as if forces associated with enclosures. The geometrical attributes of a 

of all the walls are of equal strength. grid inside of an enclosure 282 are depicted in FIG. 8. Grids 

FIG. 6c illustrates another use of enclosures in the present 60 are defined as a condition and can include parameters such 

invention. Enclosures can be used with events to provide as those below, also illustrated in FIG. 8. 

some enclosures with priority over other enclosures. The Offset (O): If the grid is inside of an enclosure, defines the 

enclosure with priority can change the characteristics of horizontal and vertical distance, in pixels, from the 

another enclosure, such as the strength of forces of the walls upper left comer of the enclosure 's deadband to the first 

of the other enclosure. For example, window 272 is asso- 65 snap point. If the grid is stand-alone, defines the 

ciated with a force enclosure 274. The walls of enclosure horizontal and vertical distance, in pixels, from the 

274 surround the borders 275 of the window and have an upper left corner of screen to the first snap point. 
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Positive Coefficient (C) : Strength of snap points (jolts). (for active application programs) and send information back 

PositiveSaturation (S): Saturation of snap points. through the translation layer to the context driver to notify 

DeadBand (DB): Vertical and horizontal spacing, in the application program. 

pixels, between snap points, , F ? r * D ^plication program to receive an event 
Grids can also be implemented as one-dimensional forces, 5 Qot * ca t tlon > the application s context must be active, i.e., he 
vjii^au ai^, ^ uu F iv.iiiwit^ uu^iLii^u^uiiai application must be active or background. Thus, inactive 
e.g.ashckmafconas^ applications will not receive event notifications. Some 
when moved. The tick marks would only be felt when the cvents ^ bc defined to have riorit over Qthcr cvents For 
mouse was moving* ithe defined dimension For example, example> tf a trigger is ^th an event m p ; 
a slider can be defined with a one-dimensional grid so that, foreground context as well as an event in a background 
as the knob of the slider is moved, a snap point is applied as context, the foreground context can be defined to have 
the knob moves over each tick of the grid. The ticks thus priority so that only the application program in the fore- 
have a spatial correlation with the distance that the knob has ground receives an event notification. Alternatively, only the 
moved and inform the user of such. background application, or both the foreground and back- 
ground applications, can receive the event notification. 
^ vents 15 Preferably, both one-time event triggers and periodic 
One structure used by the host in force feedback imple- event ^88™ are available. One time event triggers are 
mentation is called an "event." Events are notifications from triggers associated with discrete actions that have no sig- 
the force feedback device to application programs 202, 204, duration, such as breaking through an enclosure, 
and 206 of one or more actions or interactions that have m ^nodic event triggers are associated with conditions that 
« • .« i_- i • * i_- i_ t_ i_ have a finite duration, such as being inside the wall or an 
occurred in the graphical environment which has been , , , « i_ « ^ * ^ 
ji_*i_r iPjLij- n f* i enclosure, or holding down a scroll button. Dunne the 
sensed by the force feedback device 11. For example an duratkm of ^ whicfa tri ^ event * ( 

event can be any interaction of the cursor in the graphical while tfae cursor fe ^ enclosure)? the j^l repeatedly 
environment with another object. Thus, an event can be event notifications to the application program accord- 
defined when the cursor enters into an enclosure object 25 ^ g to a predetermined periodicity that can be set by the 
(described below) through a particular border. Alternatively, application program. 

an event can be triggered when the cursor moves over a As an example, an application program creates an event 

close box of a window, or the cursor moves over a file icon definition for an enclosure event. This is an event that occurs 

and a button is pressed. An event can also be defined when wn en a user moves the cursor into an enclosure. The event 

the cursor moves within a specified range of a particular 30 definition includes a unique event identifier, an identifier for 

graphical object or region. Also, an event may be defined the application program that created it, actions that trigger 

with a temporal component, e.g. an event is triggered when the event notification (entering the enclosure), and any 

the cursor remains in a particular region for a specified effects (such as the particular enclosure) which the event is 

period of time. Events are similar to button press triggers in associated with. 

that a condition is detected on the force feedback device, 3S An example of an event structure is provided below: 
independently of the host; however, the trigger for an event 
is a graphical interaction rather than a physical button press. 

An application program initially defines what the event is event 

by creating an event definition with the API 208. The event typcdcf struct { 

definition preferably includes an event identifier, an appli- 40 dword dwsize; 

cation program (or window) identifier, information specify- HWND hWndEventHandler, 

ing actions that trigger an event notification, and force dword dwfTri 

effects, if any, that the event is associated with. The appli- lpi_effect p Effect; 

cation program assigns a separate identification value to } event, 

each event at the time of creating the event, and keeps track 45 — — ^ _ _ 

of the created identification values in its own list. TTie API dwSize: Size of ^ structure in b ^ member mugt be 

can enable, disable, or delete an event at the request of an initialized before the structure is used. 

application program. hWndEventHandler: Handle of window to which event 

The translation layer receives and stores events when the notifications are sent 

application that created the event is the active or background 50 wRef: 16-bit application-defined value to identify the event, 

application program. Thus, events are preferably included in This value is defined by the application to identify events 

the active context. The translation layer preferably checks that it has created. The application assigns a separate 

for events based on the position of the cursor (or mouse) wRef value to each event at the time of creation. Each 

received from the force feedback device when the associated subsequent event notification that the application receives 

application program is (or becomes) active. The translation 55 will be tagged with its associated wRef value. When the 

layer then sends the event notification up to the context application processes the window message that receives 

driver (or API), which sends an event notification to the event notifications, it will compare the wRef value to it's 

application program, e.g. sends a message to a window own internal list and take action accordingly, 

handler which the application program checks. The event dwftrigger: Flags specifying actions that trigger the event 

notification includes the event identifier so that the applica- 60 notification. 

tion program can check its own list and identify the event. pEffect: Particular effect (e.g. enclosure), if any, in the GUI 

Once identified, the application program initiates the appro- that this event is associated with, 

priate response as defined by the developer, e.g. running The following are examples of flags that can be included as 

another application program, outputting a sound, displaying triggers for an event. 

a new image, etc. Alternatively, the force feedback device is 65 ONENTERTOP, ONENTERLEFT, etc.: The mouse broke 

loaded with the defined event similar to being loaded with into the associated enclosure by breaking through the 

effects. The local microprocessor 130 can check for events top, bottom, left, or right wall, as specified. 
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ONEXITTOP, ONEXITLEFT, etc.: The mouse broke out 
of the associated enclosure by breaking through the top, 
bottom, left, or right wall, as specified. 

ONENTERANY, ONEXITANY: The mouse broke into or 
out of the associated enclosure. 

INBOUNDTOPWALL, INBOUNDLEFTWALL, etc.: 
The mouse is in the top, bottom, left, or right wall, as 
specified, and attempting to break into the associated 
enclosure. 

OUTBOUNDTOPWAL, OUTBOUNDLEFTWALL, 
etc.: The mouse is in the top, bottom, left, or right wall, 
as specified, and attempting to break out of the asso- 
ciated enclosure. 

INB OUND ANYWALL, OUTBOUND ANYWALL: The 
mouse is in a wall and attempting to break into or out 
of the associated enclosure. 

ANYWALL: The mouse is in a wall of the associated 
enclosure. 

ONSCROLL: The user is holding down the scroll button 
of the mouse. 

ONEFFECTCOMPLEnON: The associated effect has 
completed. 

The host computer or force feedback device may also 
command the mouse 12 to act as a different type of control 
device for some embodiments. The force feedback device 11 
is primarily intended, in the embodiment described in FIG. 
2, to be used in a graphical environment such as a GUI, in 
which multi -tasking applications are provided. When the 
force feedback device is used as such, the API 208 and other 
layers described above are preferably used. However, the 
force feedback device can also be used as a different type of 
controller and may use other standard API's. For example, 
the mouse force feedback device 11 as shown in FIG. 2 may 
also be desired to be used in conjunction with game appli- 
cation programs, simulations, or the like, in which a joystick 
or other type of controller is typically used. Device 11 can 
be configured to work with such game applications in a 
"joystick mode" as if it were a standard force feedback 
joystick, or as if it were a non-force-feedback joystick. The 
user can be provided with a physical switch on the device 11 
to switch from mouse to joystick mode, or a software switch 
provided in master application 206 or other program on the 
host computer can be used to switch modes. For example, 
when the device is in the joystick mode, the host computer 
can use the DirectX API from Microsoft Corp. which 
includes force feedback functionality for standard joystick 
controllers. The device 11 becomes an absolute position 
reporting device in joystick mode. 

Mouse device 11 as shown in FIG. 2 is not easily 
applicable to many applications intended for use with a 
joystick, since the mouse is a planar device having a free 
workspace while a joystick typically has rotary degrees of 
freedom and springs to bias the joystick to a center position. 
Thus, forces are preferably used to cause the mouse to be 
more similar to a joystick. When the mouse device 11 is used 
in the joystick mode, in the preferred embodiment, spring 
forces are applied to bias the mouse 12 toward the center of 
its workspace. These forces are applied by actuators 64 and 
controlled by the local microprocessor 130 as background 
forces that are always in effect, unless the device is switched 
out of joystick mode. 

Force Feedback Device Implementations 

The force feedback interface device 11, as described 
above, preferably includes a local microprocessor 130 that 
implements various processes to provide the functionality 
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and features of the force feedback implementation described 
herein. Various aspects of the force feedback device imple- 
mentation are described below. 

Reference Frames 

The force feedback mouse device as described herein 
needs the ability to know the location of the cursor on the 
display device of the host computer. This is because the 
device must monitor the interaction of the cursor with other 

10 objects in the graphical environment to determine when 
associated forces should be output. The force feedback 
device of the present invention uses different reference 
frames to determine where the cursor is positioned. There 
are three primary reference frames that are described herein: 

15 the device (or local) frame 30, the ballistic frame and the 
screen (or host) frame 28. 

The device frame is the frame of the physical workspace 
of the force feedback device. In a preferred embodiment, the 
device frame is defined with the origin at the top left, the 

20 positive x direction to the right and the positive y direction 
down. This frame is used to describe the absolute position of 
the end point of the mechanism in its workspace. The scale 
of "movement units" for the device frame is arbitrary and 
fixed, although it should be a higher resolution than the 

25 resolution of sensors 62 to guarantee no loss of sensor data. 
Furthermore, in the preferred embodiment, the device posi- 
tion is scaled to reflect a 4:3 aspect ratio, which matches the 
preferred mechanical design (guide opening 76) as well as 
the typical computer monitor size. However, in designs 

30 having different-sized mechanical workspace, the device 
frame is scaled to the provided mechanical workspace. 

In the preferred embodiment, rate-control indexing is 
provided. This allows the mouse 12 to reach the limit to its 
workspace and still control the cursor in the same direction. 

35 In brief, the central portion of the device workspace is 
designated as the position control region, where a change in 
position of the mouse in its workspace directly correlates to 
a change in position of the cursor on the screen. The change 
in device position is translated to a change in ballistic 

40 position based on the velocity of the device, according to a 
ballistics algorithm (described below). 

An edge region of the device frame is designated as the 
rate -control region, in which the mode of moving the cursor 
changes from position control, to rate control, where the 

45 cursor continues to move on the screen in the same direction 
while the mouse is in the rate control region. The rate control 
region boundary is accompanied by a repelling force that 
pushes the user object towards the center of the device 
workspace. As the user pushes against this force the cursor 

50 is moved in a speed proportional to the distance the device 
is displaced into the rate control region. Therefore, the 
cursor moves relative to device position in the rate control 
region, not relative to device motion. This mode may be 
thought of as moving the position control central area of the 

55 device frame around the screen using rate control. This type 
of indexing is described in greater detail in co-pending 
patent application Ser. No. 08/924,462, incorporated by 
reference herein. The size of the rate control region is 
specified by a percentage. For example, a 400x300 device 

60 workspace has a rate-control region of 10%. The cursor 
position would be determined according to a position control 
paradigm when the user object position is read between the 
values of (x, y)=(20,15) and (380,285) of the device frame, 
and according to a rate control paradigm if the user object 

65 position is outside of that range. 

The screen frame is used to describe the position of the 
cursor (or other user-controlled graphical object) on the 
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screen. It is defined with the origin at the top left, the positive (or a simulated change, when in rate control mode) be added 

x direction to the right and the positive y direction down. to the ballistic frame. If changes in mouse position are 

This frame is scaled corresponding to the resolution of the always used, problems with different aspect ratios are elimi- 

display device 20 (or devices 20). For example, if a com- nated. This is described in greater detail below, 

puter monitor resolution is set to 1024x768 pixels, the x 5 Transformmg from the ballistic frame to me scr een frame 

screen position ranges from 0 to 1023 as the cursor moves fe f Qnce ^ ^ itioQ fc ^ the 

to the right and from 0 to 767 as the cursor moves down. All . . . . - , r . , 

j. • j * i i iL r c ■» , cursor position m the screen frame can be found by dividing 

coordinate communication between the force feedback . *f7 .. , . , w . / ™ % 

device and the host computer are in terms of screen coor- * e cursor posiUon by the known multiple Thus, if 

dinates (normalized for device standards). in * e °^ticframe is defined to be 10 times the resolution of 

^ t_ if ■ c t u i j j ,,\ • c t , 10 the screen frame, the ballistic position is divided by 10 to 

The ballistic frame (or scaled frame ) is a frame created , 4 . 4 . 

u *u c r ju i j ■ . a * • obtain the screen position, 

by the force feedback device to determine cursor position. v 

By applying a ballistics algorithm to the change in position Position Reporting 

of the user object, the cursor position is determined. The 

ballistic frame is defined as a multiple of the screen frame, 1S Standard mouse controllers are open-loop devices which 
i.e., the ballistic frame preferably has a higher resolution report position changes of the mouse in its workspace to the 
than the screen frame, but has the same aspect ratio as the host computer and have no ability to know the current cursor 
screen frame. The ballistic frame is created by the force position. The force feedback mouse device of the present 
feedback device so that pixel changes smaller than one invention, in contrast, needs to know the location of the 
screen pixel can be made to the cursor position, which is 20 cursor on ^ dis P la V dcvicc of lhc host computer, since the 
crucial when implementing the fast control loop rates mouse device must monitor the interaction of the cursor with 
demanded by force-feedback. For example, if the control other objects the graphical environment to determine 
frequency is 1 kHz, and a monitor provides 1000 pixels whea associated forces should be output, 
across, a change of 1 pixel per control cycle would cause the There are two primary means for a force feedback mouse 
cursor to travel the entire screen width in one second. In rate 2S to report the position of the mouse to the host and to know 
control mode, such adjustments in cursor position may have the screen position of the cursor. The first, preferred method 
to be made every control cycle, which is far too fast to allow for the mouse to know cursor location is for the mouse to 
control of the cursor. Thus, a frame with higher resolution, report (normalized) absolute coordinates to the host com- 
the ballistic frame, needs to be defined to allow the force puter to dictate the position of the cursor. For example, the 
feedback device to change cursor position at its own control 30 actual screen coordinates of cursor can be determined by the 
frequency, but which can be divided down to the screen mouse and reported to the host, and the host simply displays 
frame size to slow down the cursor as it is moved across the the cursor at the reported coordinates. This allows the mouse 
screen. In addition, the aspect ratio of the ballistic frame is to know exactly where the cursor is at all times, and to 
the same as that of the screen frame. This allows force therefore have an accurate model of the screen for use with 
effects on the device to spatially correlate with graphical 35 outputting forces. However, any ballistic behavior of the 
objects displayed on the display device. This is described in cursor must be calculated on the mouse device itself, cre- 
greater detail below. ating more calculation overhead for the local microprocessor 
The microprocessor must make transformations between 130 In addition, an absolute reporting scheme is not corn- 
frames to convert a device position to an eventual screen P atible with traditional, non-force-feedback mouse devices, 
coordinate. When transforming from the device frame to the 40 ^ excluding a default mode in case the force feedback 
ballistic frame, the microprocessor may use sensitivity. functionality is not operational. 

Sensitivity is the ratio of the size of the device frame to the A second method provides a force feedback mouse device 

size of the ballistic frame. A higher value of sensitivity that reports position changes of the mouse in its workspace, 

allows the cursor to traverse the screen with less physical as a standard mouse does today, rely on the host computer 

motion of the mouse 12, thus causing the cursor to be harder 4S to send back the cursor position. This approach allows the 

to control. A lower value of sensitivity allows more motion host software to maintain the job of doing cursor ballistics, 

on the device to cause less motion of the cursor on the but requires twice the communication as well as a need for 

screen, and is suitable for fine cursor control. At low the mouse to predict intermediate values of cursor position 

sensitivity values, the cursor may only be able to reach the while waiting for the next host cursor position update, since 

entire screen area by using rate control mode. 50 the cursor reporting occurs at a slower rate (e.g. about 50-60 

In the preferred embodiment, the cursor can exactly move Hz ) than the haptic control rate (e.g. about 1000 Hz). This 

across the entire width of the screen as the mouse 12 moves embodiment is described in greater detail with respect to 

across the extent of the position control region of the device 13. 

workspace, as long as the velocity of the mouse 12 remains The processes described below are preferably imple- 

in a middle range to provide a one-to-two ballistic mapping 55 mented as firmware in integrated circuits and memory 

(a one-to-one ballistic mapping is provided with velocities in provided on the force feedback device. Alternatively, the 

the lower range, causing less cursor movement; and a processes can be implemented as hardware or software, or a 

one-to-four ballistic mapping is provided with velocities in mixture of these. The processes themselves can be imple- 

the high range, causing more cursor movement). As sensi- mented using program instructions or code stored on or 

tivity is increased, the screen can be traversed in less than the 60 transferred through a computer readable medium, such as a 

entire position control region of the device workspace memory chip, circuit or other device; magnetic media such 

(assuming a middle range velocity). At lower sensitivities, as hard disk, floppy disk, or tape; or other media such as 

the entire screen cannot be traversed (assuming a middle CD-ROM, DVD, PCMCIA cards, etc. The program instruc- 

range velocity) without using the rate control mode, tions may also be transmitted through a channel to device 11 

Since the aspect ratios of the device frame and the ballistic 65 from a different source, 

frame may not match, updating position changes in the FIG. 9 is a block diagram illustrating a first, preferred 

ballistic frame may require that a change in device position embodiment 290 of the present invention. In this 
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embodiment, both relative and absolute position reporting of the process 300, the mouse eventually will be moved to 

modes are used. As referred to herein, relative position the physical limits of the device and the angle values will be 

reporting means that the force feedback device reports only updated accordingly. This tends to produce more accurate 

changes in position of the user object 12 to the host results than an assumed range of motion, since manufactur- 

computer, i.e. a difference between a new position and the 5 ing tolerances of the components of device 11 provide 

last reported position, while absolute position reporting slightly varying dimensions, link lengths, and angle ranges 

means that absolute coordinates are reported to the host from that cause discrepancies from device to device. In an alter- 

which the host can directly display the cursor or other nate embodiment, the force feedback device 

graphical object. Most force feedback controllers use abso- (microprocessor 130) can control the actuators of the device 

hite position reporting since the position of the cursor on the 10 to move tne mouse 12 to all the limits of the mouse 12 in a 

screen must be known to implement forces, For example, the short start-up procedure to determine the actual angle 

force feedback mouse and other interface devices described ranges. 

in U.S. Pat. Nos. 6,219,032, 5,825,308, and 6,100,874. use The normalized joint angles 306 resulting from step 304 

absolute position reporting. are processed by applying kinematics in step 308. 

One problem with absolute position reporting is that the 1S ******* equations, as is well known to those skilled in the 

host computer cannot detect the force feedback device as a art > ™ us f * l ? dcnvc * P° S1U0D ° f a f dcs * ed P°*t on a 

. j _i • . . 11 u if i • mechamcal linkage or other apparatus from known lengths 

standard input controller, such as a mouse. If relative posi- of ^ fa ^ lm | afld {h ^ &ni ]eg between ^ 

tion reporting were provided, the host computer can detect ^ Fof fc a ilion * of a int on the 

the force feedback mouse as a traditional non-force- ^ in a Cartesian (or other type) of coordinate system 

feedback mouse. This is desirable when force feedback 20 can ^ determined using geometrical equations. A current 

functionality is not currently active to allow the user to select device position 310 is determined from the kinematics of 

options within the operating system, such as during startup ste p 308, which is the position of the mouse 12 in its 

of the device before force feedback drivers have been loaded workspace . 

or during emergencies when force feedback is not enabled. xhe current device position 310 is used, with a previous 

The present embodiment provides a relative positioning 25 device position 312, to determine a device delta position 

mode when the force feedback device 11 is powered up or 314. The previous device position 312 was the current 

when the host computer 18 first detects the force feedback device position 310 in a previous iteration of process 300; 

device. In this mode, the device U provides a device delta this is indicated by the dashed line 311 between current and 

position to the host as shown in FIG. 9. The host is expecting previous device positions. The device delta position 314 is 

to receive a delta value upon startup, and thus simply 30 simply the difference between the current device position 

processes the delta position normally. This allows the user to 310 and the previous device position 312. In step 316, the 

use the device for normal postioning of the cursor in a GUI microprocessor 130 sends the determined delta position 314 

or graphical environment before force functionality has been to the host computer 18 over bus 120. This is a relative 

initialized. The microprocessor does not need to model the position reporting step and is only performed if the system 

cursor position in this stage of the process because no force 35 10 does not yet have force feedback functionality, e.g. at 

feedback functionality has yet been enabled on the device startup when the force feedback drivers are being loaded. 

11. However, once the force feedback functionality is The host computer uses the delta position to position the 

enabled, e.g., the force feedback driver software has been cursor on the screen of display device 20. For example, the 

enabled on the host computer, then an absolute position is host computer adjusts the position of a cursor by the delta 

reported to the host in place of the delta position relative 40 amount to achieve a new position of the cursor in the screen 

position reporting. In addition, ballistic parameters and the frame 28. During relative position reporting mode, the delta 

host screen size are sent to the device 11 to allow the device position 314 is sent to the host at a slower rate than the rate 

11 to determined the absolute cursor position. Force feed- of process 300; therefore, in some iterations of process 300, 

back commands are also sent to the device in this stage when the delta need not be reported to the host. This is because the 

appropriate. 45 host only needs to receive a cursor delta position at the rate 

FIG. 10 illustrates a process 300 for providing force of displaying the cursor on the screen, which typically is on 

feedback according to the preferred embodiment290 of FIG. ™e order of 60 Hz or everv 20 ms. The microprocessor 130, 

9. In a first branch of process 300, current joint angles 302 on tne other nand > must control forces at a much higher rate 

are measured by sensors of the force feedback device 11. For ( 1()00 Hz or above) and thus needs to execute process 300 

example, in the embodiment of FIGS. 1 and 2, the joint 50 milcn 

angles between members 44, 46, 48, and 50 are determined If the device*is in relative position reporting mode, the 

based on the current position of the encoder arc as sensed by process iteration is complete after the delta position 314 is 

the emitter and detector of sensor 62. The joint angles can be reported to the host. The process then returns to measure 

determined since the precise measurements of the members joint angles 302 and start the next iteration when triggered 

of linkage 40 are known and are in a known position relative 55 to do so. The microprocessor docs not need to model the 

to each other based on the sensor 62 reading. In a process cursor position in this mode because no force feedback 

step 304, the current joint angles are normalized based on functionality has yet been enabled on the device 11. 

the sensed range of the mouse 12 in its degrees of freedom. Relative position reporting mode is exited and absolute 

Since the exact range of the mouse 12 (or other user object) position reporting mode is entered when the appropriate 

may not be precisely known due to variations in the device 60 software drivers or other programs are loaded into the 

or other irregularities, the normalization process uses a memory of host computer 18 to allow the host to recognize, 

sensed range of motion to determine and assign physical interface with, and command the force feedback of device 

angles between the members of the linkage 40. For example, 11. In absolute position reporting mode, the delta position 

the minimum sensor value that has been detected so far is 314 continues to be determined as described above, 

considered to be one limit of the motion of the mouse, and 65 However, the delta position is no longer reported directly to 

the maximum sensor value detected so far is considered to the host computer, but is instead used in a ballistics process 

be the other limit to the mouse range. In successive iterations to determine a ballistic delta, described below. 
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In absolute position reporting mode, the force feedback 
device U also determines a velocity of the user object 12 for 
use in the present process in addition to the joint angle 
processing described in steps 304 and 308. A current joint 
velocity 320 is first measured. In the preferred embodiment, 5 
this is accomplished by a dedicated circuit or device that 
determines a velocity from the sensed positions reported by 
sensors 62, e.g. the sensed positions of members 44 and 48. 
For example, a "haptic accelerator" device can include a 
circuit dedicated to calculating velocity from multiple Q 
sensed positions and time periods between the sensed posi- 
tions. Such an embodiment is described in greater detail in 
U.S. Pat. No. 5,999,168, which is incorporated by reference 
herein. Alternatively, velocity sensors can be used to directly 
measure velocity of joints or members of the device. 5 

The current joint velocity 320 is used in a process step 322 
which uses kinematics in determining device velocity. As 
described above, kinematics are well-known equations for 
determining position, and also are used to determine a 
velocity of a desired point of a linkage. Thus, the kinematics 20 
can be used to determine the velocity of the mouse 12, which 
is referred to as the "device velocity" herein. The current 
device velocity 324 resulting from step 322 indicates the 
current velocity of mouse 12. 

Ballistics step 328 uses a ballistic algorithm or similar 2 s 
method to model or predict a position of the cursor in the 
screen frame 28 based on the position and velocity of the 
mouse 12. This modeling is accomplished to allow the local 
microprocessor 130 to determine where the cursor is posi- 
tioned in the ballistic frame. Using the spatial layout of 30 
graphical objects that is stored locally in memory 134 and 
which is continually updated by the host, the local micro- 
processor determines when the cursor is interacting with an 
object and when to output forces associated with the inter- 
action. The local determination of the cursor position allows 35 
the microprocessor to know the position of the cursor for the 
determination of forces. 

In the ballistics step 328, the local microprocessor uses a 
ballistics algorithm to determine a "ballistic delta" for the 
cursor, i.e. a change in position of the cursor measured in the 40 
ballistic frame. A ballistics algorithm is a method of chang- 
ing the position of the cursor so as to break the direct 
mapping between the mouse 12 position and the cursor 
position to allow the user greater control over the cursor 
position. For example, most ballistics algorithms vary the 45 
position of the cursor based on the velocity of a mouse in its 
workspace. The faster the velocity of the mouse, the greater 
the distance that the cursor is moved on the screen for a 
given distance the mouse is moved. This allows slow 
motions of the mouse to finely control the cursor, while fast 50 
motions coarsely control the cursor and allow the cursor to 
be moved quickly across the screen. Other algorithms can 
also be used which are similar to ballistics in that cursor 
position is altered to allow fine cursor control in situations 
when needed (such as for targeting graphical objects with 55 
the cursor) and to allow coarse cursor control when needed. 
Ballistics and other methods of such enhanced cursor control 
are described in greater detail in co-pending application 
08/924,462, incorporated by reference herein, and can be 
used in the present invention. 60 

The microprocessor 130 preferably uses a ballistic algo- 
rithm that provides two velocity thresholds to determine 
how far the cursor is to be moved. This is described in 
greater detail below in the method of FIG. 11. Other ballis- 
tics algorithms may be used in other embodiments; e.g., a 65 
continuous algorithm that adjusts cursor position continu- 
ously along a range of mouse velocity. 
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The microprocessor uses several types of data to deter- 
mine the ballistic delta of the cursor, including the device 
delta position 314, current device velocity 324, and host 
ballistic parameters 330. The host ballistic parameters 330 
are received by the microprocessor 130 from the host at 
some point before the process 300 starts, or when the 
ballistic model changes, and are stored in local memory 134 
to be accessed when required. Such parameters can include 
the distance to move the cursor for each threshold range of 
device velocity, the velocity values of the two thresholds, 
and any conditions concerning when to apply the ballistics 
algorithm. If the host changes ballistics algorithms, notifi- 
cation can be sent to the microprocessor 130 and new 
parameters stored in local memory 134. The microprocessor 
130 examines the current device velocity 324 and the 
parameters 330 to determine in which threshold range the 
mouse velocity currently is placed, and then multiplies the 
device delta position by the appropriate amount (e.g., 1, 2, 
or 4, depending on the velocity). This results in the ballistic 
delta 332, which is provided in terms of the ballistic frame. 
Changes in mouse position (i.e., delta positions 314) are 
used in the absolute position reporting mode of the present 
invention because this allows problems with changes to 
screen aspect ratios and resolutions to be alleviated. In 
addition, an indexing function of the device U is preferably 
implemented which may cause the ballistic delta 332 to be 
calculated differently in a rate control mode, as detailed 
below in FIG. 11. 

The microprocessor then proceeds to a process step 334, 
in which a current ballistic location (or "scaled position") 
336 of the cursor is determined in the ballistic frame. To 
accomplish this, the microprocessor uses the ballistic delta 
332, and the previous ballistic location 338. The previous 
ballistic location 338 was the current ballistic location 336 
in an earlier iteration of process 300, as indicated by dashed 
line 337. Previous location 338 can be stored in local 
memory 134 and then discarded after being used to deter- 
mine the current ballistic location. At its simplest level, the 
current ballistic location 336 is determined in step 334 by 
taking the previous ballistic location 338 and adjusting that 
location by the ballistic delta 332. 

The current ballistic location 336, however, cannot be 
directly transformed to a screen pixel position in the screen 
frame. Step 342 uses the current ballistic location 336 and 
the host screen size 340 to determine the screen pixel 
location. 

The host screen size 340 is the current pixel resolution 
and/or aspect ratio being displayed on the display device 20 
of the host computer 18. For example, the host may be 
displaying a screen resolution of 800x600 pixels, 1024x768 
pixels, or 1280x960 pixels. Also, in some embodiments, the 
host may be displaying a different aspect ratio. For example, 
a typical computer monitor aspect ratio is 4:3; however, if 
two monitors are connected to provide a continuous screen 
space, the aspect ratio becomes 8:3. i.e., double the size of 
a single screen in one dimension. The host screen size 340 
can be received by the device 11 when the ballistic param- 
eters 330 are received, or at any other time (and/or when 
sensitivity information is received, if applicable). Different 
resolutions or aspect ratios have different numbers of pixels 
in respective dimensions of the screen and thus cause 
displayed objects to appear in different areas of the screen. 
For example, if a graphical object is centered in the screen 
at 640x480 resolution, the object will appear in the upper 
left quadrant of the screen when the resolution changes to 
1280x960. 

The change in resolution or aspect ratio can create a 
problem with force sensations created by the force feedback 
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device. The device has been sent a spatial layout of objects host receives the screen pixel location 346 and displays the 

from the host that indicates the coordinates of an object on cursor appropriately. 

the screen. When the screen size is changed, and the device In 344 ti, e local processor 130 use s the (preferably 

* not informed, the device will output forces corresponding non . norm alized) screen pixel location (and other data) to 

to the object as ,t appeared in the old screen size. Tins can s determine whether forces ^ out t ^ out „ 

cause large discrepancies between where an object is dis- e r . , , . r , r , 

i j j u fL r c i_- * , * • those forces. For example, the local microprocessor checks 

played and where the forces of an object are encountered in t . t . , , . - *7 . . 4 . • . « 

the device workspace. For example, if an icon is at coordi- the spaUd layout ofgraptacal objects which is stored m ocal 

nates (64,48) on a 640x480 resolution screen, the device mem .°7 U4 , dete ™™* whether the screen pixel loca- 

i \u \ *u * * jincr P4 L *• aon intersects or contacts any of those objects. If so, there 

knows that the icon is positioned 10% of the entire screen 1f) , , . . , . r 4 . . t : J , ' 

dimension away from the top and left borders of the screen 10 ™* a k f P rc , c ™* tl ? ct, ° n ' ^P"*""* ° n 

workspace. The device accordingly outputs associated f P^ al f* ccl "bisected. Such a force would be 

forces when the mouse is moved into tte region of the " 1 ™ Iat . cd ^ on a variety of data. Such da a may include 

device workspace corresponding to the icon, I*. 10% of the * c ^ ™1<*«Y™> the d ev ,ce posUion 310. the elapsed 

workspace distance from the top and left borders of the 1S ta °? e ? ?"». » eve ?f> * e th . stance & om a dlffere ( ? t g"P hlcal 

workspace. However, after a resolution change to 1280x960, " ***** m *» ^ hlcai environment, a history of past mouse 

a. • • j* i j fflt f*L j • • r motion, etc. Some examples of forces that can be output in 

the icon is displayed 5% of the screen dimension away from graphical environment are described in U S Pat Nos 

the top and left borders. The device, however, would output * oof™ k ^ eSCnbed m t V , N ° S * 

f . a *u u f 1 - V * « , ^ 6,219,032 and 5,825,308, both incorporated by reference 

forces based on the old position of the icon, which no longer herrin^ 

corresponds with the position of the icon. Thus, the device 20 

needs to be told the new screen resolution so it may provide ° Dce ste P 346 * complete (and any other desired steps in 

forces at the appropriate region of the device workspace. me provision of forces, which is not detailed herein), the 

Similarly, a change in aspect ratio can cause distortion in $™^ when ™f ed «*»™ * nwi ™S of j° int ^ 

the force feedback sensations. For example, a circle object ™ d J om f velocilv 320 to ^ m another lteratloa of the 

is displayed on a 4:3 aspect ratio screen and has a force wall 25 absolute P ositlon re P ortin g Process, 

surrounding its shape. The circle will feel like a circle on the No error correction is required in the present embodiment, 

device if the ballistic frame also has an aspect ratio of 4:3. unlike the embodiment below, because the force feedback 

However, if another monitor is activated so that the screen device 11 is operating in absolute mode when the pixel 

has an 8:3 aspect ratio, the circle object will feel like an oval position is determined The host is no longer receiving 

due if the screen is directly mapped to the device frame, i.e. 30 simply a change in position (delta) of the user object, but is 

if the device frame is "stretched" to fit the screen frame. The receiving an absolute coordinate in a coordinate system and 

use of delta positions 314 prevents this distortion. Since only nee ^ perform no other significant processing (such as 

changes in the mouse position are used to determine cursor ballistics) to display the cursor, i.e., the local microprocessor 

position, and not an absolute mouse position in the device solely performs the necessary adjustments, such as ballistics, 

workspace, the cursor is positioned without distortion. 35 and reports the adjusted position to the host computer. Thus, 

However, the new aspect ratio still needs to be communi- me nosl tocal microprocessor have the same position 

cated to the force feedback device since there is a larger area valu e for the cursor, and no opportunity exists for the two 

in which the cursor may be positioned. For example, in a 4:3 positions to diverge. 

aspect ratio, the microprocessor might clip a pixel location FIG. 11 is a flow diagram illustrating a method 360 of the 
to the edge of the screen if that location were beyond the 40 present invention for implementing enhanced cursor control 
edge of the displayed range. However, on an 8:3 aspect ratio, and indexing in the absolute position reporting mode of 
the location should not be clipped since the displayed range method 300. This method provides ballistics for cursor 
is larger. positioning as well as indexing functionality, and many of 
To prevent distortion caused by a change in screen size, me steps of method 360 overlap those of method 300. As 
the ballistic location 336 of the cursor is divided by a divisor 45 described above, the mouse workspace preferably includes 
that takes the host screen size into account. For example, if a central position control region in which the mouse controls 
the ballistic frame is 10 times the resolution of the screen me cursor according to a position control scheme. An 
frame, then the divisor is 10: the ballistic location 336 would isometric edge region is provided around the position con- 
be divided by 10 to achieve a screen pixel position of the trol region of the mouse workspace. When the mouse is 
cursor if the host screen size had not changed. If the host 50 moved into the isometric region, a resistive force is applied 
screen size did change, then the ballistic frame is resized to to me mouse and the cursor is moved according to a rate 
take the new screen size into account. Preferably, the screen control method, where the speed of the cursor is proportional 
pixel position is equal to the screen size times the ballistic to me amount of penetration into the isometric region. This 
location divided by the ballistic frame size (i.e. the divisor). allows a user to move the cursor throughout the screen frame 
Thus, if screen resolution on the host has doubled, the 55 without causing the user to experience disconcerting inter- 
divisor would halve, and the screen pixel location would ruptions due to the mouse colliding with physical limits to 
double. The host can also send sensitivity information to the me mouse workspace. This indexing method is described in 
device. If a new sensitivity value is received, the ballistic greater detail in co-pending patent application Ser. No. 
frame is resized accordingly; as sensitivity is increased, the 08/924,462. 

ballistic frame size is decreased. 60 The method 360 is similar to the indexing methods 

Once the screen pixel location is translated from the described above in that a position control region and iso- 

ballistic position 336, the screen pixel location is then metric edge region are provided in the mouse workspace, 

typically normalized to a standardized range of numbers Method 360 is intended to be used on the preferred embodi- 

since communication software and devices have standard ment 2SH) of FIGS. 9 and 10 and accounts for the use of a 

values to receive. The normalized screen pixel position 346 65 ballistic frame on the mouse and for changes in screen size, 

is then sent to the host computer as an absolute position in Method 360 begins at 362, in which raw sensor data is 

step 350, e.g., as coordinates for display on the screen. The read using the sensors 62. In step 364, the joint positions of 
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linkage 40 are calculated from the raw sensor data. In step 
366, position kinematics are used to determined the device 
position (position of the mouse 12 or other user object) in the 
device frame, as explained above with reference to FIG. 10. 
The last device position is stored in local memory 134 in 
step 368, and a delta device position is determined in step 
370 using the current device position resulting from step 366 
and the last device position stored in step 368. From step 
366, step 372 is initiated, which checks whether the device 
position is in the position control region of the device frame. 
If not, the mouse is in the isometric region and is in rate 
control mode, where a resistive force is preferably output on 
the mouse opposing movement into the region. In step 374, 
the rate control delta is calculated to be the difference 
between the device position and the rate control edge; this 
provides the distance of penetration into the isometric 
region. In step 376, the rate of cursor movement is equal to 
the maximum rate times the rate control delta divided by the 
rate control edge width (i.e., the width of the isometric 
region). The rate control delta divided by the rate control 
edge width provides the percentage of penetration of the 
mouse into the isometric region, which is designated to be 
the same percentage of the maximum rate of the mouse. The 
maximum rate can be set by the developer; for example, 
1000 ballistic pixels per second or 8000 ballistic pixels per 
second. In step 378, the ballistic position (i.e. the position of 
the cursor in the ballistic frame) is set equal to the old 
ballistic position plus the rate determined in step 376. The 
process then continues to step 394, detailed below. 

If the device position is in the position control region in 
step 372, then a position control method is followed. Steps 
384—392 implement a preferred ballistics algorithm to pro- 
vide enhanced cursor control to the user of device 11. This 
algorithm is generally as follows. If the current device 
velocity is below the first threshold, the cursor is moved by 
the smallest amount, e.g., a one-to-one mapping between 
device movement units and ballistic pixels. If the current 
device velocity is between the first and second thresholds, 
the cursor is moved double the amount of pixels as it is 
moved when mouse velocity is under the first threshold, e.g. 
a one-to-two mapping between device and ballistic frames. 
If the user object velocity is above the second threshold, the 
cursor is moved double the amount of pixels as between the 
first and second thresholds (a one-to-four mapping between 
device and ballistic frames). In other embodiments more or 
less thresholds, or a continuous function, can be used. Also, 
the host may change the ballistics algorithm in a variety of 
ways and indicate the changes to the device 11 using ballistic 
parameters 330 as described for FIG. 10. 

Thus, step 384 checks the device velocity so that a 
ballistic position may be determined, the device velocity is 
determined in steps 380 and 382, where step 380 determines 
joint velocities from the sensor data read in step 362 and 
device velocity is determined from the joint velocities and 
velocity kinematics in step 382, as described above in FIG. 
10. In step 384, this device velocity is compared to the 
ballistic thresholds used in the ballistics algorithm. If the 
device velocity is within the first ballistic threshold in step 
384, then step 386 determines the new ballistic position to 
be the old ballistic position phis the delta device position 
(from step 370). The process then continues to step 394, 
detailed below. If the device velocity is greater than the first 
ballistic threshold, then in step 388 the device velocity is 
compared to the second ballistic threshold. If the device 
velocity is between the first and second thresholds, then in 
step 390 the new ballistic position is set equal to the old 
ballistic position plus twice the delta device position. This 
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causes the cursor to move at a faster rate, based on mouse 
velocity. The process then continues to step 394, detailed 
below. If the device velocity is greater than the second 
ballistic threshold in step 388, then in step 392 the new 
5 ballistic position is set equal to the old ballistic position plus 
4 times the delta device position. The process then continues 
to step 394. 

In step 394, the screen position of the cursor is set equal 
to the ballistic position as determined in one of steps 378, 

10 386, 390, and 392 divided by a divisor. The divisor is known 
from the resolution of the ballistic frame and the resolution 
of the screen frame, as follows. In step 396, the screen 
resolution (or size) is obtained from the host. In step 398, the 
divisor is set equal to the ballistic range (or resolution) 

15 divided by the screen resolution as detailed above. 

After the screen position is determined in step 394, it is 
sent to the host computer in step 399 and used in the display 
of the cursor in the screen frame. The device 11 also 
determines and outputs forces, as detailed in FIG. 10. It 

20 should be noted that the forces on mouse 12 may be 
determined or calculated differently in rate control 
(isometric) mode than in position control mode, as described 
in greater detail in co-pending application Ser. No. 08/924, 
462. 

25 

FIG. 12 is a block diagram illustrating a second embodi- 
ment 400 of the present invention for reporting positions to 
a host computer from a force feedback device. Embodiment 
400 provides only relative position reporting from the force 

30 feedback device 11 to the host computer 18. This is advan- 
tageous in that the force feedback mouse device is detected 
by the host as a standard non-force-feedback mouse, where 
the host computer can use standard drivers and other soft- 
ware. This allows the user to control GUI functions even 

35 when force feedback is not currently active, such as during 
startup of the device or during failures of the force feedback 
functionality. In addition, a relative position reporting pro- 
cess is more appropriate than an absolute position reporting 
process for such interface devices as trackballs, in which an 

^ unrestricted workspace and range of motion is provided for 
the user object 12. For example, the spherical ball in a 
trackball controller can be moved in any direction for any 
number of revolutions and is not as suitable for use with an 
absolute coordinate system as a device having a workspace 

AC with known limits. 

45 

Embodiment 400 provides a device delta position to the 
host computer from which the host computer determines a 
cursor screen pixel position. The host computer sends cursor 
position updates to the force feedback device to provide the 

50 cursor position which the device needs to determine when 
forces are output and to determine the force characteristics. 
However, since the host cursor position updates occur at a 
much slower rate, the device must model the cursor position 
on its own to be able to output forces between host cursor 

55 updates. Thus, the host sends ballistic parameters and screen 
size information so the device can model the cursor position. 
When the device receives an update, it corrects for any error 
between its modeled position and the actual cursor position, 
as detailed below. The host also sends force feedback 

5 0 commands to the device indicate which forces are to be 
output. 

FIG. 13 illustrates a process 410 for implementing the 
second embodiment 400 of FIG. 12. A significant difference 
between method 300 and method 410 of the first embodi- 
es ment is that the host independently determines a screen 
cursor position in the present embodiment in parallel with 
the microprocessor modeling the cursor screen position. 



06/17/2003, EAST Version: 1.04.0000 



US 6,3' 

41 

Since the host controls the actual position of the cursor, its 
screen position is considered the "correct" or actual cursor 
position, while the device modeled value is only approxi- 
mate. The host must continually send the actual screen 
position to the device so that errors in the modeled screen 
position can be corrected and so that the forces can be 
correctly correlated with actual cursor position. 

Process 410 includes many similar steps to process 300. 
The current joint angles 302 are measured and normalized in 
step 304, and the normalized joint angles 306 are processed 
with kinematics in step 308 to provide a current device 
position 310. The current device position 310 and previous 
device position 312 are used to determine a delta position 
314, which is reported to the host computer 18 in step 316 
over bus 120. As in process 300, this portion of the process 
410 functions like a traditional relative mouse, where only 
a delta value is reported to the host computer. However, 
unlike the process 300, the delta position 314 is always 
reported to the host computer, i.e., there is no absolute 
position to report at any stage of the process. The host 
computer uses the delta position to position the cursor or 
other user-controlled graphical object on the display device 
20. For example, the host computer adjusts the position of a 
cursor using the delta amount and its own ballistics algo- 
rithm to achieve a new position of the cursor in the screen 
frame 28. The delta position 314 is sent to the host at a 
slower rate than the rate of process 410; therefore, in some 
iterations of process 410, the delta need not be reported to 
the host. This is because the host only needs to receive a 
cursor delta position at the rate of displaying the cursor on 
the screen, which typically is on the order of 60 Hz or every 
20 ms. The microprocessor 130, on the other hand, must 
compute forces at a much higher rate (1000 Hz or above) 
and thus needs to execute process 300 much faster. 

The current joint velocity 320 also is read in process 410 
(both the joint angles 302 and the joint velocity 320 are 
always measured in the current process). Kinematics are 
applied in step 322 to obtain the current device velocity 324, 
and ballistics are applied in step 328 using the device 
velocity 324, ballistic parameters 330, and delta position 314 
to obtain a "modeled" pixel delta 332. The pixel delta 332 
is a modeled or predicted value in this embodiment because 
the host has determined the actual screen position using its 
own ballistics algorithm and using the delta position 314 
provided by the device. The microprocessor 130 preferably 
uses in step 328 the same ballistics algorithm used by the 
host computer in its determination of cursor position. 

In step 412, a current pixel location is modeled using the 
modeled pixel delta 332, a previous pixel location 416, and 
the host screen size 340, resulting in a current pixel location 
414. Step 412 is similar to step 334 of process 300 in that 
pixel delta 332 and the previous pixel location 416 are used 
substantially similarly. The host screen size information can 
be used to determine whether the determined pixel location 
should be clipped or not. For example, if the aspect ratio has 
changed from 4:3 to 8:3, double the amount of area is 
provided. If the pixel location is one that would normally 
extend beyond the edge of a 4:3 screen area, it would be 
clipped to the border of the 4:3 screen, but may not be 
clipped if it is still within the 8:3 area. In one embodiment, 
if the change in screen size is to a different resolution, the 
current pixel location can also be adjusted depending on the 
change in resolution. For example, if screen resolution has 
been adjusted from 640x480 to 1280x960, then the pixel 
delta can be doubled before it is added to the previous pixel 
location. 

Step 412 is different from step 334 in FIG. 10 in that a 
current pixel error 418 is included in the determination of the 
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current pixel location 414. The pixel error 414 is the error 
between the cursor position modeled by the local micropro- 
cessor and the actual cursor position as determined and 
displayed by the host computer. Although the microproces- 

5 sor is typically applying the same ballistics algorithm and 
screen size adjustments to the cursor pixel location as the 
host computer, errors can be introduced to cause the two 
cursor location calculations to diverge. For example, in 
some embodiments, the host computer can "clip" the 

10 received delta position 314, i.e., ignore the delta position so 
that the cursor stays at a constant position. Such a technique 
is used, for example, to provide realistic obstruction force 
sensations when forces are limited in magnitude, as 
described in greater detail in U.S. Pat. No. 6,028,593. Or, 

15 error can be caused by an application program on the host 
computer moving the cursor based on the program's own 
criteria and completely independently of the force feedback 
device. Or, error can be caused by the host computer 
switching to a different ballistics algorithm, where a delay 

20 exists between the host using the new ballistic algorithm and 
the force feedback device receiving the parameters for the 
new algorithm. 

The current pixel error 414 is determined by examining 
the current pixel location 414 at the time of reporting the 

25 delta position 314 and a current screen position of the cursor 
that has been received from the host computer. The host 
computer periodically reports a current screen position 420 
of the cursor to the microprocessor 130, where the screen 
position is based on the delta position 314 reported to the 

30 host in step 316. The reporting of the screen position by the 
host is triggered by receiving the delta position 314 from the 
device, and the microprocessor recalculates pixel error when 
it receives the host screen position 420. 
The microprocessor compares the received screen posi- 

35 tion 420 with the modeled current pixel location 414 at the 
time of report 316 (it should be noted that the current pixel 
location 414 may have a previously-determined error cor- 
rection value included in its determination, as explained 
below). If the two locations are the same, no error exists 

40 between the two cursor positions, and the microprocessor 
uses the current pixel location 414 as the location of the 
cursor in its force calculations and force outputs. If the two 
locations are not the same, error exists between the two 
cursor positions and the current pixel location 414 must 

4S therefore be corrected. 

An error correction value is determined as the difference 
between the current pixel location 414 and the host screen 
position 420. This value, when added to (or subtracted from) 

50 the pixel location 414, will cause the two cursor locations to 
be the same. Alternatively, if the microprocessor knows the 
error, then the current pixel location can be adjusted to 
provide no error. 
The screen position 414 from the host is reported to the 

55 microprocessor 130 at a slower rate than the rate of process 
410, since the force feedback process must be much faster 
than the displaying processes of the host computer. There 
may therefore be several iterations of process 410 before the 
screen position 420 is reported by the host to the micro-pro- 

50 cessor 130. Therefore, once the error correction value is 
determined after one report of screen position 420, that same 
value is continually used in iterations providing the deter- 
mination of current pixel location 414 until the next screen 
position 420 is received from the host. 

65 In the preferred embodiment, it is desirable to "smooth 
out" the correction of the error in pixel location 414 by only 
incrementally changing the pixel location 414 in each itera- 
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lion of process 410. For example, if the current pixel 
location 414 were adjusted with the entire error correction 
value at one time, this might cause a large change in the 
pixel location 414. If forces output on the user object are 
based on the current pixel location 414 of the cursor, then the 
user may notice the correction of the error by feeling a small 
jump or discontinuity in the forces applied to mouse 12 after 
the error is corrected. This is because the forces were 
determined based on a pixel location having a large error at 
one point in time, and then the forces are based on the 
corrected pixel location at the next point in time. Thus, the 
difference in pixel locations in each iteration is preferably 
smoothed by only partially correcting the current pixel 
location 414. With such smoothing, the user does not feel as 
large a discontinuity in forces when the error in pixel 
position 414 is corrected. The error correction rate is pref- 
erably based on the desired system modeling. The error can 
be corrected at a rate so that there is no error between the 
pixel location 414 and the host screen position 420 by the 
time a new host screen position 420 is received (which may 
provide a new error, of course). Or, the error can be 
corrected at a rate that spans multiple received host screen 
positions 420. Typically, the correction of errors is desired to 
be as slow as possible to maintain high fidelity of forces to 
the user, yet the error must be corrected fast enough so as not 
to create a discontinuity between what the host is displaying 
on the display device and the forces the user feels on the 
force feedback device. 

FIG. 14 is a diagrammatic illustration of an example of 
the pixel location error correction process explained above. 
The line 426 represents a time scale, and each vertical mark 
represents another iteration of process 410. At the first 
iteration, the pixel location 414 (PL) is determined to be 192 
(assuming a single dimension of user object motion) and the 
error 418 between the pixel location and host screen position 
is 0. At the next iteration, the pixel location is 200, and the 
delta position of 4 is reported to the host as in step 316. Over 
the next iterations, the pixel location is determined to be 200, 
201, and 202. At the sixth iteration, the pixel location is 
determined to be 203; in addition, the host reports a host 
screen position of 201. Since this screen position was 
determined based on the delta position 314 reported to the 
host in the second iteration, the error is measured between 
the pixel location determined at the second iteration (200) 
and the host screen position received at the sixth iteration 
(201). The error value of 1 is then included in the current 
pixel location, so that the pixel location of 203 is corrected 
to be 204 based on the error value. This example demon- 
strates that the pixel location 384 determined by the micro- 
processor 130, when error exists, is corrected a few itera- 
tions behind the actual screen position determined by the 
host. The microprocessor 130 locally stores the pixel loca- 
tion 414 that was determined in the iteration when the delta 
position 314 was reported to the host so that the error 
correction value can be accurately determined. 

It should be noted that, in the example of FIG. 14, the 
error value determined at the sixth iteration continues to be 
added indefinitely. Thus, if further errors are indicated at 
later host reports, the new error value must be added to the 
old error value. FIG. 15 is a diagrammatic illustration of the 
error correction value increasing over time. Initially the error 
correction value is 0, then 1. Over the next host report of the 
screen position, the current error is 2 pixels, which is added 
to the previous error correction value of 1 so that the total 
error correction value is 3. Likewise, in the next host report, 
the current error is 3 pixels, so that the total error correction 
value is 6. In one embodiment, if a very large error 



30,936 Bl 

44 

accumulates, the microprocessor can make the error zero in 
one iteration regardless of the effect on the user. 

FIG. 15 also demonstrates the smoothing or incremental 
correction of the modeled pixel location in the present 

5 invention. An error of 2 pixels exists at the iteration 428. The 
value of 2 is not added to the previous error value of 1 
immediately, since that may cause a force discontinuity to 
the user. Instead the 2 pixels are added gradually to the error 
correction value over the next few iterations. For example, 

10 the actual value (AV) added to the pixel location 414 is 0 for 
the first two iterations after iteration 428, then AV»2 for 
three more iterations, until the full value of 2 is added in 
following iterations. 
Referring back to FIG. 13, once the current pixel location 

15 414 is determined, the local processor 130 uses that location 
in step 424 to determine whether any forces should be output 
and outputs those forces. It should be noted that most forces 
are determined preferably based on the velocity of the user 
object in its workspace, not the velocity of the cursor in the 

20 graphical environment. Thus, these forces will always feel 
the same to the user, regardless of any error between the 
modeled pixel location 414 and the host screen position 420, 
since they is independent of screen velocity. However, some 
forces are dependent on interactions in the graphical 

25 environment, and will be affected by a cursor position error. 
Likewise, the ability to initiate or discontinue forces based 
on interactions in the graphical environment is also affected 
by cursor position error. 

30 Once step 424 is complete (and any other desired steps in 
the provision of forces, which is not detailed herein), the 
process preferably returns to the measuring of joint angles 
302 and joint velocity 320 to begin another iteration of the 
process. 

35 Force Feedback Features as Implemented in the 
Present Invention 

In the above described embodiments, the host computer 
performs such tasks as the implementation and display of the 

40 graphical (or other) environment and cursor, the implemen- 
tation of multi-tasking application programs, the sending of 
high-level force feedback commands to the force feedback 
device, and the reception of position (or other) data from the 
force feedback device indicating the user's desired interac- 

45 tion with the computer-implemented environment The force 
feedback device, in contrast, performs such tasks as reading 
the position of the user object, receiving high level force 
feedback commands from the host computer, determining 
whether a force should be output based on conditions sent 

50 from the host computer, and implementing and outputting 
the required forces when necessary. 

Some tasks, however, may be implemented in either the 
host computer or the force feedback device, and there are 
different trade-offs to choosing one implementation over the 

55 other. Events, for example, are one such feature. Another 
force feedback feature that can be implemented in either the 
host or the force feedback device is "clipping." Clipping is 
the selective omission of reported position data when dis- 
playing the cursor or other user-controlled graphical object 

60 to cause a sensory effect on the user in certain circumstances. 
Such circumstances include those where displaying the 
cursor in correlation with the precise user object position 
would not provide a desired effect. The most common case 
for clipping is when the cursor encounters a surface or object 

65 in the graphical environment which is desired for the user to 
experience as a hard obstruction or wall. The wall is desired 
to prohibit further movement into the surface or object; 
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however, due to limitations in hardware, a strong enough 
force cannot be output on the user object to actually prevent 
motion into the wall. Tnus, clipping is performed: the user 
object (e.g., mouse) is allowed to be moved by the user into 
the wall against a resistive force, but the reported position 5 
data indicating this movement is clipped or ignored so that 
the cursor is displayed positioned against the wall surface 
where it first encountered the wall. The user sees the cursor 
prevented from moving into the wall and feels a resistive 
force, and believes that it is a hard obstruction even though Q 
the user object can actually be moved into the wall; this is 
because the user heavily relies on the visual perception of 
the interaction. Clipping is described in greater detail in U.S. 
Pat. Nos. 6,028,593 and 5,825308. 

Clipping can be easily performed by the force feedback 1S 
device to ease computational burden on the host. For 
example, the local microprocessor 130 can check whether 
the user object is being moved into a wall in the graphical 
environment; if so, the microprocessor can simply discard 
the position data and report a constant position (or delta) to 20 
the host computer that would cause the host to display the 
cursor against the wall surface. This allows the host com- 
puter to be completely ignorant of the clipping process and 
simply display the cursor at the reported position, thus 
freeing the host to perform other tasks. However, problems 25 
occur in this implementation due to extra burden on the local 
microprocessor. Since the microprocessor is implementing a 
very fast force feedback loop in which the user object 
position is read and forces are output, it is very disruptive to 
this loop when the microprocessor has to check whether to 30 
clip the read position of the user object, and may cause a 
small distortion in output forces. 

The host computer 18 can also perform clipping in an 
alternate embodiment Since it is desirable to keep the 
application programs 202 and 204, the operating system, and 35 
other high level programs ignorant of the clipping process, 
a lower level program preferably handles the clipping. For 
example, the translation layer 218 as shown in FIG. 4 (or 
alternatively the context driver or API) can check for the 
conditions that cause clipping to be applied. If clipping is to 40 
be performed, the translation layer alters the input position 
data appropriately before it is sent on to the context driver, 
API, operating system and any application programs. Prob- 
lems with this implementation include increased load on the 
host with the overhead of intercepting and translating 45 
incoming messages. 

A different force feedback feature that can be imple- 
mented either by the force feedback device or the host 
computer is "pressure clicks" or "click surfaces." As 
described above, these are surfaces, objects or regions in a 50 
graphical environment which have additional functionality 
based on the position of the cursor in relation to the object. 
Thus, a border of a window can be designated a click surface 
such that if the user overcomes a resistive force and moves 
the cursor (or user object) over a threshold distance into the 55 
border, a function is activated. The function can be scrolling 
a document in the window, closing the window, expanding 
the window size, etc. A similar click surface can be used on 
an icon or button to select or activate the icon or button. 

The force feedback device can implement click surfaces 60 
to ease computational burden on the host computer. The 
local microprocessor can check whether the cursor has 
moved into an click surface or region and output the resistive 
force as appropriate. When the cursor is moved past the 
threshold distance, the microprocessor reports to the host 65 
computer that the action associated with the click surface 
has been made; for example, the microprocessor reports that 
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a button press or double click has been performed by the 
user The host receives a signal that a button has been 
pressed and acts accordingly. Or, the microprocessor reports 
that a particular click surface has been specifically selected, 
where the host can interpret the selection and implement an 
associated function. 

The host may also implement at least a portion of the 
functionality of click surfaces. The microprocessor can 
output the resistive (or other type) force associated with the 
click surface, but only reports the user object position to the 
host. When the host determines that the cursor (or user 
object) has moved beyond the threshold distance, the host 
implements the function associated with the click surface 
(scrolling text, closing a window, etc.) As with clipping, it is 
preferred that the translation layer handle this functionality 
by modifying data accordingly before passing it to the 
context driver and API. 

While this invention has been described in terms of 
several preferred embodiments, it is contemplated that 
alterations, permutations and equivalents thereof will 
become apparent to those skilled in the art upon a reading of 
the specification and study of the drawings. For example, 
although examples in a GUI are described, the embodiments 
herein are also very well suited for other two-dimensional 
graphical environments and especially three-dimensional 
graphical environments, where a user would like fine posi- 
tioning in manipulating 3-D objects and moving in a 3-D 
space. For example, the isometric limits are quite helpful to 
move a cursor or controlled object in a 3-D environment 
further than physical limits of the interface device allow. 

In addition, many different types of forces can be applied 
to the user object 12 in accordance with different graphical 
objects or regions appearing on the computer's display 
screen and which may be mouse-based force sensations or 
cursor-based force sensations. Also, the various features of 
the embodiments herein can be combined in various ways to 
provide additional embodiments of the present invention. In 
addition, many types of user objects and mechanisms can be 
provided to transmit the forces to the user, such as a mouse, 
trackball, joystick, stylus, or other objects. Furthermore, 
certain terminology has been used for the purposes of 
descriptive clarity, and not to limit the present invention. It 
is therefore intended that the following appended claims 
include all such alterations, permutations, and equivalents as 
fall within the true spirit and scope of the present invention. 

What is claimed is: 

1. A method for interfacing a multi-tasking graphical 
environment implemented on a host computer with a force 
feedback interface device coupled to said host computer, 
wherein multiple application programs are simultaneously 
running in said multi-tasking environment, wherein one of 
said multiple application programs is active and the other 
application programs are inactive, the method comprising: 
receiving force effects from each of at least two of said 
multiple application programs simultaneously running 
in said multi-tasking graphical environment, each of 
said force effects describing a force sensation that is 
output by said force feedback interface device; 
storing said force effects into contexts allocated in 
memory of said host computer, wherein each of said 
contexts is associated with only one of said at least two 
application programs running on said host computer, 
and wherein each of said force effects is stored in said 
context associated with said application program that 
sent each of said force effects; 
of said force effects in said contexts of said at least two 
application programs, providing only said force effects 
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in said context of said active application program to be 
stored in said force feedback interface device such that 
only said force effects received from said active appli- 
cation program are output to a user of said force 
feedback interface device; 5 

receiving instructions from each of said at least two 
application programs, said instructions instructing said 
force feedback interface device to start an output of at 
least one of said force effects sent from each of said 
application programs; and 1Q 

of said instructions from said at least two application 
programs, sending only said instructions from said 
active application program to said force feedback inter- 
face device such that only said force effects from said 
context of said active application program are output to 15 
said user of said force feedback interface device. 

2. A method as recited in claim 1 wherein said instructions 
include commands, and wherein commands from inactive 
running application programs to start or stop an output of 
force effects are ignored and not sent to said force feedback 20 
interface device. 

3. A method as recited in claim 2 wherein when said active 
application program becomes inactive and a different one of 
said at least two application programs becomes active, new 
force effects are sent to said force feedback interface device 2S 
to replace said force effects sent to said force feedback 
interface device for said previously active application 
program, said new force effects being included in a context 
associated with said different application program. 

4. A method as recited in claim 1 further comprising: 30 
sending force effects in a context associated with a 

background application program to said force feedback 
interface device regardless of whether said background 
application program is active or inactive in said multi- 
tasking environment; and 35 

sending instructions from said background application 
program to said force feedback interface device such 
that said force effects from said background application 
program are output to said user of said force feedback 
interface device, 40 

such that said force feedback interface device may output 
forces based on said force effects from said active 
application program and from said background appli- 
cation program. 

5. A method as recited in claim 1 wherein said contexts 45 
further include at least one event that can be triggered by 
said force feedback device, wherein said event includes 
conditions based on interactions of a cursor in said graphical 
environment such that when said conditions are met, an 
event notification is sent to an application program associ- 50 
ated with said event. 

6. A method as recited in claim 1 wherein said force 
effects include a spring, a damper, and a vibration. 

7. A method as recited in claim 1 wherein said multi- 
tasking graphical environment includes a Windows™ oper- 55 
a ting system environment 

8. A method as recited in claim 7 wherein said force 
effects are created as effect objects in an application pro- 
gramming interface in said Windo™ operating system 
environment, and wherein said effect objects are sent to said 60 
force feedback interface device when created by said active 
application program. 

9. A method as recited in claim 1 wherein said force 
effects sent to said force feedback interface device are stored 

in memory of said force feedback interface device and 65 
output as force sensations when commanded to be output by 
said active application program. 



10. A method as recited in claim 1 wherein at least one of 
said force effects describes a force sensation that provides an 
enclosure, said enclosure including at least one wall having 
an inner wall portion and an outer wall portion, each of said 
wall portions associated with different forces felt by said 
user from said device. 

11. A method as recited in claim 10 wherein said enclo- 
sure is an ellipse. 

12. A method as recited in claim 10 wherein said enclo- 
sure is a rectangular object including four sides, each of said 
sides providing said inner wall and said outer wall. 

13. A method as recited in claim 12 wherein a magnitude 
of force sensations associated with top and bottom sides of 
said enclosure is greater than a magnitude of force sensa- 
tions associated with left and right sides of said enclosure so 
that a user can perceive all of said sides as having equal 
magnitude when using said force feedback interface device. 

14. A method as recited in claim 10 wherein at least one 
of said force sensations associated with said enclosure is 
adjusted after a user-controlled cursor is moved into said 
enclosure, wherein a particular one of said application 
programs is informed of said movement into said enclosure 
by using an event stored in said context associated with said 
particular application program and sent to said force feed- 
back interface device. 

15. A method as recited in claim 10 wherein said enclo- 
sure is used to control force sensations associated with other 
enclosures defined within the interior space of said enclo- 
sure. 

16. A computer readable medium including program 
instructions executable by a host computer, said program 
instructions interfacing a multi-tasking graphical environ- 
ment implemented on said host computer with a force 
feedback interface device coupled to said host computer, 
wherein multiple application programs are simultaneously 
running in said multi-tasking environment, wherein one of 
said plurality of application programs is active and the other 
application programs are inactive, the program instructions 
performing: 

receiving force effects from each of at least two of said 
multiple application programs simultaneously running 
in said multi-tasking graphical environment, each of 
said force effects describing a force sensation that is 
output by said force feedback interface device; 

storing said force effects into contexts allocated in 
memory of said host computer, wherein each of said 
contexts is associated with only one of said at least two 
application programs running on said host computer, 
and wherein each of said force effects is stored in said 
context associated with said application program that 
sent each of said force effects; 

of said force effects in said contexts of said at least two 
application programs, providing only said force effects 
in said context of said active application program to be 
stored in said force feedback interface device; 

receiving instructions from each of said at least two 
application programs, said instructions instructing said 
force feedback interface device to start an output of at 
least one of said force effects sent from said at least two 
application programs; and 

of said instructions from said at least two application 
programs, sending only said instructions from said 
active application program to said force feedback inter- 
face device, said instructions causing only said force 
effects from said context of said active application 
program to be output to said user. 
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17. A computer readable medium as recited in claim 16 
wherein when said active application program becomes 
inactive and a different one of said application programs 
becomes active, new force effects are sent to said force 
feedback interface device to replace said force effects sent to 
said force feedback interface device for said previously 
active application program, said new force effects being 
included in a context associated with said different applica- 
tion program. 

18. A computer readable medium as recited in claim 16 
further comprising performing a step of sending force effects 
in a context associated with a background application pro- 
gram to said force feedback interface device regardless of 
whether said background application program is active or 
inactive in said multi-tasking environment, such that said 
force feedback interface device may always output forces 
based on said force effects from said active application 
program or from said background application program. 

19. A computer readable medium as recited in claim 18 
wherein only said background application program may 
send said force effects to said force feedback interface 
device when inactive. 

20. A computer readable medium as recited in claim 16 
wherein said contexts further include at least one event that 
can be triggered by said force feedback device, wherein said 
event includes conditions based on interactions of a cursor 
in said graphical environment such that when said condi- 
tions are met, an event notification is sent to an application 
program associated with said event. 

21. A computer readable medium as recited in claim 16 
wherein said force effects include a spring, a damper, and a 
vibration. 

22. A computer readable medium as recited in claim 16 
wherein said multi-tasking graphical environment includes 
an operating system environment. 

23. A computer readable medium as recited in claim 22 
wherein said force effects are created as effect objects in an 
application programming interface in said operating system 
environment, and wherein said effect objects are sent to said 
force feedback interface device when created by said active 
application program. 

24. A computer readable medium as recited in claim 16 
wherein said force effects sent to said force feedback inter- 
face device are stored in memory of said force feedback 
interface device and output as force sensations when com- 
manded to be output by said active application program. 

25. A computer readable medium as recited in claim 16 
wherein said force effects describe an enclosure force 
sensation, said enclosure force sensation providing at least 
one wall having an inner wall and an outer wall, each of said 
walls associated with said enclosure force sensation. 

26. A method for interfacing a multi-tasking graphical 
environment implemented on a host computer with a force 
feedback interface device coupled to said host computer, 
wherein a plurality of application programs having force 
feedback functionality are simultaneously running in said 
multi-tasking environment, wherein only one of said appli- 
cation programs is active, the method comprising: 

enabling a reception of force commands from at least two 
of said application programs having force feedback 
functionality, said force commands characterizing 
force effects to be output by said force feedback inter- 
face device; 

enabling a storage of each of said force effects in one of 
a plurality of contexts, each of said contexts being 
allocated in memory of said host computer, and each of 
said contexts being associated with only one of said at 
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least two application programs running on said host 
computer, and wherein each of said force effects is 
stored in said context associated with said application 
program that sent said force effect; 
5 enabling a determination of which one of said at least two 
application programs sending said force commands is 
currently active in said multi-tasking environment; and 
of said force effects in said contexts of said at least two 
application programs, enabling a sending of only said 
10 force effects in said context of said active application 
program to said force feedback interface device, said 
sent force effects being available to be output to said 
user by said force feedback interface device. 
27. A method as recited in claim 26 further comprising 
is enabling a reception of instructions from said at least two 
application programs that sent said force commands to 
start an output of at least one of said force effects sent 
from said application programs; and 
enabling a sending of only said instructions from said 
20 active application program to said force feedback inter- 
face device, said instructions causing only said force 
effects from said context of said active application 
program to be output to said user of said force feedback 
interface device. 
25 28. A method as recited in claim 27 wherein if said 
application program providing said instructions is inactive, 
said instructions to cause said force effects to be output are 
ignored. 

29. A method as recited in claim 27 wherein said sent 
30 force effects include force effect characterization data that 

are stored in memory of said force feedback interface 
device. 

30. A method as recited in claim 29 wherein said instruc- 
tions include a start command to start the output of a force 

35 effect stored in a context, wherein said force effect is 
commanded to be output on said force feedback interface 
device only if said application program providing said start 
command is active. 

31. A method as recited in claim 26 further comprising 
40 receiving a stop command from one of said at least two 

application programs to stop the output of a force effect, 
wherein said interface device is commanded to stop said 
output of said force effect only if said application program 
providing said start command is active. 

45 32. A method as recited in claim 26 further comprising 
enabling a background force feedback master application to 
run in said graphical user interface simultaneously with said 
plurality of application programs, said background master 
application providing force feedback to graphical objects in 

50 said graphical environment. 

33. A method as recited in claim 32 wherein commands 
and force effects from said background master application 
are always sent to said force feedback interface device, 
regardless of whether said background application is active, 

55 such that said force feedback interface device outputs forces 
based on said force effects from said active application 
program and from said background application program. 

34. A method as recited in claim 26 wherein said force 
commands include a command to set a device state structure. 

60 35. A method as recited in claim 34 wherein said device 
state structure includes a force gain level for all forces output 
by said force feedback interface device. 

36. A method as recited in claim 26 wherein said active 
application program is a first application program, and 

65 further comprising receiving an indication that said first 
application program has become inactive and that a second 
one of said running application programs has become active, 



06/17/2003, EAST Version: 1.04.0000 



US 6,300,936 Bl 



51 



52 



10 



wherein further force commands from said first application 
program are not sent to said interface device and force 
commands from said second application program are sent to 
said interface device while said second application program 
is active. 5 

37. A method as recited in claim 26 wherein any com- 
mands from an inactive running application program to start, 
stop, or pause said output of force effects by said interface 
device are ignored and are not sent to said interface device. 

38. A method as recited in claim 26 wherein said contexts 
further include data characterizing at least one event that can 
be triggered by said force feedback device, wherein said 
event includes conditions based on interactions of a cursor 
in said graphical environment such that when said condi- 
tions are met, an event notification is sent to an application 
program associated with said event. 15 

39. A method for controlling a force feedback interface 
device coupled to a host computer displaying a graphical 
environment and a cursor within said graphical environment 
on a display screen, the method comprising: 

reading sensor data and determining a position of a user 20 
maoipulandum in a device frame of said force feedback 
interface device from said sensor data, said manipu- 
landum being physically contacted and moveable by a 
user, 

determining a device delta position of said manipulandum 25 
that includes the change in position of said manipulan- 
dum from a previous position, and reporting said 
device delta position to said host computer when said 
host computer is not enabled to receive an absolute 
position of said cursor from said force feedback inter- 30 
face device; and 

when said host computer is enabled to receive said 
absolute position of said cursor from said force feed- 
back interface device, 

determining said absolute position from said device 35 
delta position, said absolute position provided in a 
screen frame of said display screen, 

reporting said absolute position to said host computer 
to allow said host computer to display said cursor in 
said screen frame at a position derived from said 40 
absolute position, and 

using a scaled position of said manipulandum related to 
said absolute position in determining a force to be 
output by actuators of said force feedback interface 
device. 45 

40. A method as recited in claim 39 wherein said scaled 
position of said manipulandum is determined from said delta 
position, and wherein said absolute position is determined 
based on said scaled position. 

41. A method as recited in claim 40 wherein said scaled 50 
position is determined based on a ballistic delta position, 
said ballistic delta position being derived from said delta 
position using a ballistics process. 

42. A method as recited in claim 41 wherein said ballistics 
process uses a velocity of said manipulandum in said device 55 
frame in said determination of said ballistic delta position. 

43. A method as recited in claim 42 wherein, in deter- 
mining said ballistic delta position, said ballistics process 
multiplies said device delta position by one if said velocity 

of said manipulandum is below a first threshold velocity, by 60 
two if said velocity of said manipulandum is between said 
first threshold and a second threshold velocity, and by four 
if said velocity of said manipulandum is above said second 
threshold velocity. 

44. A method as recited in claim 41 further comprising 65 
receiving ballistic parameters from said host computer, said 
ballistic parameters characterizing said ballistics process. 



45. A method as recited in claim 40 further comprising 
receiving a current screen resolution of said display screen 
of said host computer to allow said microprocessor to adjust 
said absolute position based on said screen resolution. 

46. A method as recited in claim 40 further comprising 
receiving a current screen aspect ratio of said display screen 
of said host computer to allow said microprocessor to adjust 
said absolute position based on said screen aspect ratio. 

47. A method as recited in claim 39 wherein said scaled 
position of said manipulandum is a ballistic position in a 
ballistic frame, said ballistic frame having a resolution that 
is a multiple of a resolution of said screen frame. 

48. A method as recited in claim 39 wherein said scaled 
position is determined according to a rate control method for 
indexing when said manipulandum is moved within a pre- 
determined distance of a limit to said device frame. 

49. A method as recited in claim 39 further comprising 
outputting said force on said manipulandum. 

50. A method as recited in claim 39 wherein said delta 
position is determined from a current manipulandum posi- 
tion and a previous manipulandum position, wherein said 
current and previous manipulandum positions are deter- 
mined from measured angles of joints of said force feedback 
device. 

51. A method as recited in claim 39 wherein said host 
computer is not enabled to receive said absolute position 
when said host computer has first detected said force feed- 
back interface device, and wherein said host computer is 
enabled to receive said absolute position after force- 
feedback related driver software has been loaded into 
memory of said host computer. 

52. A force feedback interface device coupled to a host 
computer displaying a graphical environment and a user- 
controlled graphical object within said graphical environ- 
ment on a display device, the interface device comprising: 

a user manipulandum physically contacted by a user and 
movable in a planar workspace in two degrees of 
freedom; 

a sensor that provides raw data describing a device 

position of said manipulandum in said workspace; 
a plurality of actuators coupled to said manipulandum that 

provide a force on said manipulandum in said planar 

workspace; and 
a local microprocessor, separate from said host computer 

and coupled to said sensor and to said actuators, said 

local microprocessor: 

determining a delta position of said manipulandum that 
includes the change in position of said manipulan- 
dum from a previous position, said microprocessor 
reporting said delta position to said host computer 
when said host computer is not enabled to receive an 
absolute position of said user-controlled graphical 
object to allow said host computer to determine a 
position at which to display said user-controlled 
graphical object; and 

when said host computer is enabled to receive said 
absolute position of said user-controlled graphical 
object, determining said absolute position of said 
user-controlled graphical object from said delta 
position, reporting said absolute position to said host 
computer to allow said host computer to display said 
user-controlled graphical object at a position derived 
from said absolute position, and using said absolute 
position in determination of said force output by said 
actuators if said force is based at least partially on an 
interaction of said user-controlled graphical object 
with another graphical object in said graphical envi- 
ronment. 
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53. A force feedback interface device as recited in claim 
52 wherein said absolute position is determined from a 
ballistic position, said ballistic position being determined 
from said delta position by using a ballistic algorithm, and 
wherein said force is determined using said ballistic posi- 5 
tion. 

54. A force feedback interface device as recited in claim 

52 wherein said local microprocessor receives a current 
screen size of said display screen of said host computer to 
allow said microprocessor to adjust said absolute position 10 
based on said screen size. 

55. A force feedback interface device as recited in claim 

53 wherein said local microprocessor receives ballistic 
parameters from said host computer which characterize said 
ballistic algorithm. 15 

56. A force feedback interface device as recited in claim 
52 wherein said local microprocessor modifies said absolute 
position for indexing when said manipulandum and said 
user-controlled graphical object position have become offset 
and when said manipulandum is moved within a predeter- 2 q 
mined distance of a limit to said device workspace. 

57. A force feedback interface device as recited in claim 
52 wherein said manipulandum is a mouse. 

58. A force feedback interface device as recited in claim 
52 wherein said host computer is not enabled to receive said ^ 
absolute position when said host computer has first detected 
said force feedback interface device, and wherein said host 
computer is enabled to receive said absolute position after 
force-feedback related driver software has been loaded into 
memory of said host computer. 30 

59. A method for controlling a force feedback interface 
device coupled to a host computer displaying a graphical 
environment and a cursor within said graphical environment 
on a display screen, the method comprising: 

reading sensor data and determining a position of a user 35 
manipulandum in a workspace of said force feedback 
interface device from said sensor data, said manipu- 
landum being physically contacted and moveable by a 
user; 

determining and reporting a position of said manipulan- 40 
dum to said host computer, said position used by said 
host computer to display said cursor in said graphical 
environment; and 

receiving information from said host computer describing 
a current screen size of said display screen of said host 45 
computer, wherein said current screen size is included 
in said determination of said position reported to said 
host computer. 

60. A method as recited in claim 59 wherein said position 

is a change in position of said manipulandum in said 50 
workspace. 

61. A method as recited in claim 59 wherein said position 
is an absolute position at which to display said cursor. 

62. A method as recited in claim 59 wherein said screen 
size is a current screen resolution of said display screen. 55 

63. A method as recited in claim 59 wherein said screen 
size includes information describing a current aspect ratio of 
said display screen. 

64. A method as recited in claim 59 further comprising 
determining on said force feedback interface device a force 60 
to be output by at least one actuator of said force feedback 
interface device, wherein said force is determined based at 
least in part on said position reported to said host computer. 

65. A method as recited in claim 64 wherein said position 
reported to said host computer is a screen position at which 65 
to display said cursor, and wherein said determining of said 
position includes scaling a manipulandum position in accor- 
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dance with said current screen size to determine said screen 
position for said current screen size. 

66. A method as recited in claim 65 wherein said scaling 
said device position includes scaling a ballistic position 
based on said manipulandum position and on a velocity of 
said manipulandum in said workspace, and wherein said 
screen position is determined from said ballistic position. 

67. A method of interfacing a tactile feedback mouse with 
a software application running on a host computer, said 
method comprising: 

providing a physical object contacted by a user and 
moveable by said user on a planar surface; 

providing a sensor that tracks motion of said physical 
object with respect to said planar surface, wherein 
motion data provided by said sensor is transmitted to 
said host computer for updating a status of a graphical 
object on a graphical display device, said motion data 
transmitted over a communication bus; 

providing an actuator that delivers a tactile sensation to 
said user when said user is in contact with said physical 
object, said actuator delivering said tactile sensation in 
response to tactile signals received from said host 
computer over said communication bus; and 

enabling two modes of operation, said two modes being 
a relative mode wherein said motion data is reported to 
said host computer as relative data, and an absolute 
mode wherein said motion is reported to said host 
computer as absolute data. 

68. A method as recited in claim 67 further comprising 
providing a switch on said tactile feedback mouse, said 
switch able to select each of said two modes. 

69. A method as recited in claim 67 wherein said host 
computer can select each of said two modes. 

70. A method as recited in claim 67 wherein said relative 
mode is a mouse mode for reporting data when said force 
feedback device is a mouse, and wherein said absolute mode 
is a joystick mode for reporting data when said force 
feedback device is a joystick. 

71. A method as recited in claim 67 wherein one of said 
modes is active at any one time, said active mode being said 
relative mode when said software application is a general 
application program running on said host computer and 
controlling results of interactions between said graphical 
object and other displayed objects, and said active mode 
being said absolute mode when said software application is 
a force feedback game running on said host computer and 
controlling said graphical object. 

72. A method as recited in claim 71 wherein said force 
feedback games run on said host computer under a 
Microsoft® DirectX® API. 

73. A method as recited in claim 67 wherein said relative 
data is used to provide position control input within said 
software application and said absolute data is used to 
provide rate control input within said software application. 

74. A method as recited in claim 67 wherein said relative 
data is interpreted by said software application as mouse 
motion and said absolute data is interpreted by said software 
application as joystick motion. 

75. A method of interfacing a tactile feedback mouse with 
a software application running on a host computer, said 
method comprising: 

providing a physical object contacted by a user and 
moveable in a planar workspace on a planar surface; 

providing a sensor that tracks motion of said physical 
object with respect to said planar surface, wherein 
motion data provided by said sensor is transmitted to 
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said host computer for updating a location of a graphi- 
cal object on a graphical display device, said motion 
data transmitted over a communication bus; 

providing an actuator that delivers a tactile sensation to a 
palm of said user when said palm is in contact with said 
physical object, said actuator delivering said tactile 
sensation in response to tactile signals received from 
said host computer over said communication bus; and 

providing said tactile feedback mouse with capability of 
reporting said motion data to said host computer as 
relative data and reporting said motion data to said host 
computer as absolute data, said relative data represent- 
ing an incremental change in position of said physical 
object in said planar workspace, and said absolute data 
indicating an absolute position of said physical object 
in said planar workspace. 

76. A method as recited in claim 75 wherein said capa- 
bility of reporting said motion data includes two modes of 
operation such that one of said modes is active on said tactile 
feedback mouse, said two modes being a relative mode 
wherein said motion data is reported to said host computer 
as relative data, and an absolute mode wherein said motion 
is reported to said host computer as absolute data. 

77. A tactile feedback mouse device interfacing with a 
software application running on a host computer, said mouse 
device comprising: 

a physical object contacted by a user and moveable by 
said user on a planar surface; 

at least one sensor that tracks motion of said physical 
object with respect to said planar surface, wherein 
motion data provided by said sensor is transmitted to 
said host computer for updating a status of a graphical 
object on a graphical display device, said motion data 
transmitted over a communication bus; 
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an actuator operative to output a tactile sensation received 
by said user when said user is in contact with said 
physical object, said actuator o inputting said tactile 
sensation in response to tactile signals received by said 
mouse device from said host computer over said com- 
munication bus; and 

a local processor electrically coupled to said at least one 
sensor and to said actuator, said processor providing a 
reporting mode on said mouse device, wherein said 
reporting mode can be either of two modes, said two 
modes including a relative mode wherein said motion 
data is reported to said software application as relative 
data, and an absolute mode wherein said motion is 
reported to said software application as absolute data. 

78. A tactile feedback mouse device as recited in claim 77 
further comprising a switch on said tactile feedback mouse, 
said switch able to select which of said reporting modes is 
active. 

79. A tactile feedback mouse device as recited in claim 77 
wherein said host computer can select said relative mode or 
said absolute mode. 

80. A tactile feedback mouse device as recited in claim 77 
wherein said reporting mode is said relative mode when said 
software application is a general application program run- 
ning on said host computer, and wherein said reporting mode 
is said absolute mode when said software application is a 
force feedback game running on said host computer. 

81. A tactile feedback mouse device as recited in claim 77 
wherein said relative data is interpreted by said software 
application as mouse motion and said absolute data is 
interpreted by said software application as joystick motion. 
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